---
kind: concept
slug: language-model-pretraining
title: Language model pretraining
summary: >-
  Train a Transformer on unlabeled data with a self-supervised objective, then reuse it by fine-tuning, prompting, freezing or as a component. Covers XLNet, PaLM, Qwen, French, Dutch, multilingual, byte-level, materials, chemistry, dialogue and knowledge-graph models, frozen transfer, word budgets and corpus effects. Trade-off: objective and data versus compute and use.
tags: [pretraining, language-modeling, masked-language-modeling, transformer, scaling, domain-adaptation, embeddings, factuality, fairness, multilingual, dialogue, knowledge-graph, data-budget, open-weights, mixture-of-experts, retrieval, code, data-quality, fine-tuning, tables, documentation, monolingual, small-models, surveys, protein, document-links, arabic, replaced-token-detection, quantization, scientific-text, motion, listener-motion, tool-use, dutch, byte-level, materials-science, chemistry]
papers: ["1906.08237", "2204.02311", "1911.03894", "1912.05372", "1910.00883", "1907.09538", "2211.05100", "2004.07180", "2006.08218", "2203.01054", "2003.01200", "2105.05727", "2111.02840", "2201.10005", "2206.04624", "2207.01206", "2211.08411", "2305.08283", "2310.10196", "2311.00176", "2610.00223", "2103.05247", "2109.14739", "2201.06642", "2204.07580", "2210.09338", "2504.08165", "2302.13971", "2205.01068", "2112.06905", "2112.04426", "2304.01373", "2012.02110", "2305.10403", "2403.08295", "2404.14219", "2308.12950", "2101.00027", "2012.13255", "2107.13586", "2005.08314", "2104.08758", "2111.01243", "2306.11644", "2308.02976", "2402.06196", "2203.15827", "2206.13517", "2303.17568", "2103.11943", "2309.16609", "2101.01785", "2111.09543", "2210.02414", "2211.09085", "2305.06161", "2306.14795", "2307.06435", "2406.12793", "2308.10897", "2005.07503", "2112.07916", "2311.16079", "2606.25495", "2007.13184", "2202.10447", "1912.09582", "2109.15290", "2202.11176", "2209.01712"]
related: [linear-attention, text-summarization, text-classification, sentiment-analysis, local-global-attention, experience-replay, self-consistency, fill-in-the-middle, weight-quantization, vector-quantization, permutation-language-modeling, relative-positional-encoding, multi-query-attention, model-flops-utilization, chain-of-thought-prompting, dataset-curation, contrastive-learning, domain-adaptation, masked-language-modeling, in-context-learning, self-supervised-learning, metric-learning, transfer-learning, graph-neural-network, semi-supervised-learning, sequence-labeling, decoding-strategies, hallucination-mitigation, retrieval-augmented-generation, instruction-tuning, rlhf, tool-use, llm-agent, imitation-learning, adversarial-robustness, benchmark-validity, algorithmic-fairness, llm-serving, time-series-forecasting, multi-task-learning, dialogue-generation, knowledge-graph-completion, multilingual-evaluation, curriculum-learning, knowledge-distillation, prompt-engineering, parameter-efficient-fine-tuning, named-entity-recognition, mixture-of-experts, code-completion, positional-encoding, preference-optimization, intrinsic-dimension, llm-ethics, llm-evaluation, neural-scaling-laws, subword-tokenization, table-question-answering, extractive-question-answering, multi-hop-reasoning, nucleus-sampling, post-training-quantization, conditional-random-field, toxicity-detection]
status: draft
updated_at: 2026-11-01
---

## Overview

Pretraining fits a network to a large corpus of raw data with an objective that needs no labels, so the learned representations or the generator can later be reused on many tasks. Three decisions define a recipe:

1. **Objective.** A left-to-right (autoregressive, AR) objective maximizes the product of next-token conditionals; each position then only sees one side of its context [arxiv:1906.08237 §2.1]. A denoising autoencoder (AE) objective in the style of BERT ([arXiv:1810.04805](https://arxiv.org/abs/1810.04805)) reconstructs masked tokens from both sides, but treats the masked tokens as independent of each other and uses a `[MASK]` symbol that never appears at fine-tuning time [arxiv:1906.08237 §2.1]. A permutation LM keeps the exact AR factorization but over a random order, so every token is trained with context from both sides in expectation [arxiv:1906.08237 §2.2]. A survey files these under *generative* objectives; ELECTRA's replaced-token detection, which trains a discriminator on a small generator's outputs, is placed in a *generative-contrastive* family [arxiv:2006.08218 §2, §5.3]. The same survey reports that contrastive pretraining for NLP was unconvincing at its March 2021 version, and that NLP contrastive work mostly sits in fine-tuning [arxiv:2006.08218 §4.4].
2. **How the model is consumed.** Either fine-tune the encoder on each task (XLNet's setting, with a separate fine-tuning recipe per task [arxiv:1906.08237 tab. 8]) or prompt a causal decoder with zero, one or a few examples and no gradient updates (PaLM's setting [arxiv:2204.02311 §1, §6.1]). For aspect-based sentiment analysis, later work reformulates the task to suit a pretrained model (sentence-pair classification, reading comprehension, sequence generation) instead of only swapping in its embeddings [arxiv:2203.01054 §5].
3. **Data and scale.** Corpus, parameters, tokens, optimizer stability and hardware efficiency. PaLM studies what happens when parameter count is pushed to 540B on one epoch of 780B tokens [arxiv:2204.02311 §1, §3]. CamemBERT studies how much and which kind of web text a single-language encoder needs [arxiv:1911.03894 §1, §6]; FlauBERT trains a larger, cleaned French mix of 71 GB [arxiv:1912.05372 §3.1].

The invariant across the masked and AR variants: labels come from the data itself, and the training signal is a likelihood or reconstruction loss over items the model did not see. XLNet stresses one part of this explicitly: the permutation lives only in the attention masks, never in the token order [arxiv:1906.08237 §2.2]. SPECTER's labels are citation links between papers, which are also part of the data, and its loss ranks a positive paper above a negative one rather than predicting tokens [arxiv:2004.07180 §2.3].

The same recipe also serves as a component of larger systems: an encoder that supplies node features to a graph model [arxiv:2105.05727 §3.1], an embedder trained on paired text [arxiv:2201.10005 §2.2], and a language-model backbone for a web agent [arxiv:2207.01206 §4.2]. Other cards study what the pretraining data does to the resulting model, including which facts it recalls and which political leanings it carries, and what a training run costs in energy and carbon [arxiv:2211.08411 §3.1] [arxiv:2305.08283 §4.1] [arxiv:2610.00223 §5.1].

## Variants

### Permutation language modeling (XLNet)

Sample a factorization order z per sequence and maximize the AR likelihood along z [arxiv:1906.08237 eq. 3]. Because a standard parameterization cannot tell which position it is asked to predict, XLNet uses two-stream self-attention: a content stream (ordinary self-attention including the token) and a query stream that knows the target position but not its token and produces the logits [arxiv:1906.08237 §2.3, eq. 4]. Only the last ≈1/K tokens of z are targets (K = 6 for Large), because the full objective converged slowly [arxiv:1906.08237 §2.3, tab. 7]. The backbone is Transformer-XL with relative positional encodings, segment-level memory and relative segment encodings [arxiv:1906.08237 §2.4, §2.5]. At fine-tuning the query stream is dropped and the model is a normal Transformer-XL [arxiv:1906.08237 §2.3].

Reported effects (author-reported): with the same data (BooksCorpus + Wikipedia) and BERT's hyperparameters, XLNet-Large beats the best of three BERT-Large variants on all 11 compared benchmarks, e.g. RACE 77.4 vs 75.1 and RTE 81.2 vs 74.0 [arxiv:1906.08237 tab. 1]. At Base size, the ablation separates the gains: a DAE objective on a Transformer-XL backbone already beats BERT-Base (RACE 65.03 vs 64.3), and the permutation objective adds more (66.66 with K = 6) [arxiv:1906.08237 tab. 6]. Removing memory hurts most on RACE, the long-context task; adding next-sentence prediction does not reliably help [arxiv:1906.08237 §3.4, tab. 6]. Scaled to 32.89B tokens, XLNet reaches 85.4 on RACE test against RoBERTa's 83.2 [arxiv:1906.08237 §3.1, tab. 2].

Cost: XLNet-Large trained for about 5.5 days on 512 TPU v3 chips (500K steps, batch 8192, length 512) and was still underfitting [arxiv:1906.08237 §3.1, tab. 7]. Several ingredients (span-based target sampling, the "bidirectional data pipeline", memory length) are described only loosely [arxiv:1906.08237 §3.1].

Card: [XLNet: Generalized Autoregressive Pretraining for Language Understanding](/papers/1906.08237).

### Replaced token detection with shared embeddings (DeBERTaV3)

A small generator fills masked positions, and a discriminator labels every token as original or replaced [arxiv:2111.09543 §3.1] [eq. 3] [eq. 4]. Unlike masked language modeling, which trains on about 15% of tokens, the discriminator is trained on every token [arxiv:2111.09543 §2.3.1]. The generator has half the discriminator's depth, and the two losses are joined with weight 50 [arxiv:2111.09543 §3.1] [§2.3.2]. The variant's contribution is how the two models share token embeddings. The discriminator reads the generator's embedding with its gradient stopped and adds a zero-initialised residual that only it trains, which the authors call gradient-disentangled embedding sharing (GDES) [arxiv:2111.09543 §3.3].

Reported effects (author-reported, 125k-step ablation): GDES reaches 89.3 on MNLI-m against 88.8 for plain sharing and 88.3 for no sharing [arxiv:2111.09543 tab. 2]. On GLUE dev, DeBERTaV3-large averages 91.37, against 90.00 for DeBERTa-large and 89.46 for ELECTRA-large [arxiv:2111.09543 tab. 3]. Caveats: the Section 3 ablations and the Section 4 full runs use different data, batch sizes and step counts [arxiv:2111.09543 §3.1] [§4]; the released code defaults to λ = 10 and its scripts use other settings than the paper [code: DeBERTa/apps/tasks/rtd_task.py#L392]; the GLUE and SQuAD numbers are dev-set results with the fine-tuning grid chosen on the same dev sets [tab. 11] [§6.4]. Card: [DeBERTaV3](/papers/2111.09543).

### Dense causal decoder at scale (PaLM)

A decoder-only Transformer trained with mean next-token log-likelihood plus a small z-loss ($10^{-4} \cdot \log^2 Z$) [arxiv:2204.02311 §5], in three sizes (8B, 62B, 540B) on the same 780B-token mixture, one epoch, no repeated subcorpus [arxiv:2204.02311 §3, tab. 1]. Architecture changes versus a vanilla Transformer: SwiGLU MLP, parallel attention and MLP in each layer (about 15% faster training at large scale, small quality loss at 8B, none at 62B), multi-query attention, RoPE, tied embeddings, no biases, and a lossless 256k SentencePiece vocabulary that splits numbers into digits [arxiv:2204.02311 §2].

Training recipe [arxiv:2204.02311 §5]: unfactored Adafactor with parameter scaling, LR $10^{-2}$ for 10k steps then decayed as $1/\sqrt{k}$, $\beta_2 = 1 - k^{-0.8}$, global-norm clipping 1.0, weight decay $lr^{2}$, no dropout, 2048-token packed sequences, and a batch ramp from 512 to 2048 sequences for 540B. Data: 50% social-media conversations, 27% quality-weighted web pages, 13% books, 5% GitHub code, 4% Wikipedia, 1% news [arxiv:2204.02311 tab. 2]. The 540B run had about 20 loss spikes; the fix was to rewind about 100 steps and skip 200–500 batches, which relies on a bitwise-deterministic, step-indexed data pipeline [arxiv:2204.02311 §5, §5.1].

Systems: 2 × 3072 TPU v4 chips, 12-way model × 256-way fully sharded data parallelism inside each pod, 2-way data parallelism across pods, no pipeline parallelism [arxiv:2204.02311 §4]. The paper defines model FLOPs utilization (MFU) and reports 46.2% for PaLM 540B against 21.3–32.5% for GPT-3, Gopher and Megatron-Turing NLG [arxiv:2204.02311 §4.1, tab. 3].

Reported effects (author-reported): few-shot SOTA on 28 of 29 English NLP tasks [arxiv:2204.02311 §6.1, tab. 4], 69.3 on 5-shot MMLU against 67.5 for Chinchilla [arxiv:2204.02311 tab. 6], and 58% on GSM8K with 8-shot chain-of-thought plus a calculator [arxiv:2204.02311 tab. 10]. Gains are not smooth in scale: on about 25% of 150 BIG-bench tasks the 540B score exceeds a log-linear extrapolation from 8B→62B by more than 10 points [arxiv:2204.02311 §6.2]. Training the 62B model on to 1325B tokens raised its MMLU from 53.7 to 62.8, so the 62B model was not data-saturated at 780B tokens [arxiv:2204.02311 tab. 6, app. F].

Card: [PaLM: Scaling Language Modeling with Pathways](/papers/2204.02311).

### Whole-word masked encoder for one language (CamemBERT)

CamemBERT retrains RoBERTa's recipe on French, with two changes: a SentencePiece tokenizer instead of WordPiece, and whole-word masking [arxiv:1911.03894 §4.3]. The model is BERT-BASE (12 layers, hidden 768, 110M parameters), with a 32k vocabulary learned from 10⁷ sampled sentences [arxiv:1911.03894 §4.2, §4.3]. The objective is MLM with dynamic masking and no next-sentence prediction. Whole-word masking samples 15% of the words and makes every subword piece of a sampled word a target, so no piece of a word can reveal the rest of it [arxiv:1911.03894 §4.3]. Optimization is Adam (β₂ = 0.98), 10k warmup steps to a peak LR of 7e-4, polynomial decay, 100k steps at batch 8192 sequences of at most 512 tokens, on 256 V100 GPUs for about a day [arxiv:1911.03894 §4.3]. The authors note that performance was still rising when they stopped [arxiv:1911.03894 §4.3].

Reported effects (author-reported, fine-tuned, best of 4 or 10 runs selected on validation): French XNLI 82.5 at 110M parameters, against 80.1 for XLM-R BASE at 270M [arxiv:1911.03894 tab. 4]; UPOS 98.18 and LAS 92.57 on UD GSD [arxiv:1911.03894 tab. 2]. Whole-word masking adds about 0.5 XNLI points and little elsewhere [arxiv:1911.03894 app. A, tab. 8]. Data is the French part of OSCAR (138GB). At equal size, Common Crawl-derived text (OSCAR, CCNet) beat 4GB of Wikipedia, with the largest gaps on the spoken treebank and XNLI [arxiv:1911.03894 §6.1, tab. 5]. The 4GB OSCAR subset came close to the full corpus except on NER, where 138GB was about 0.9 F1 better [arxiv:1911.03894 §6.2, tab. 5]. These comparisons hold at a fixed 100k-step budget, which is about 400 passes over 4GB, and validation loss was still falling [arxiv:1911.03894 §6.2]. The LARGE model, trained on CCNet, gains on NLI and NER but not on tagging or parsing [arxiv:1911.03894 app. B, tab. 8]. Results cover French only [arxiv:1911.03894 §1].

Card: [CamemBERT: a Tasty French Language Model](/papers/1911.03894).

### Base and large French encoder on a larger mixed corpus (FlauBERT)

FlauBERT is a BERT-style encoder trained with masked language modeling only, in two sizes: base (12 layers, hidden 768) and large (24 layers, hidden 1024) [arxiv:1912.05372 §3.2, tab. 1]. The corpus is 71 GB of cleaned French text, about 12.8 billion tokens, from 24 sub-corpora taken from WMT19, OPUS and Wikimedia [arxiv:1912.05372 §3.1, tab. 10]. Common Crawl is the largest part; it was cut from 215 GB to 43.4 GB by aggressive cleaning that drops very short sentences and repetitive or non-meaningful content [arxiv:1912.05372 §3.1].

Optimization uses Adam (β₁ = 0.9, β₂ = 0.98, weight decay 0.01) with an effective batch of 8192 sequences. Warmup is 24k steps to a peak of 6e-4 for base, and 30k steps to 3e-4 for large. The paper does not state how the rate decays after warmup [arxiv:1912.05372 §3.2]. The authors report about 410 GPU-hours on 32 V100 GPUs for base and 390 hours on 128 GPUs for large [arxiv:1912.05372 §3.2]; the implied totals of about 13,120 and 49,920 GPU-hours are derived here and not stated in the paper. Large uses pre-norm and the paper describes layer dropping, but that sentence is ambiguous about which sizes it covers [arxiv:1912.05372 §3.2].

Reported effects (author-reported, fine-tuned on FLUE): both sizes beat multilingual mBERT on most tasks [arxiv:1912.05372 §5]. Large has the top score on book classification (95.00 vs 92.30 for CamemBERT) and on verb sense disambiguation (50.48 F1) [arxiv:1912.05372 tab. 3, tab. 8]. CamemBERT is ahead on PAWS-X paraphrase (90.14 vs 89.49 for FlauBERT-base and 89.34 for large) [arxiv:1912.05372 tab. 4]. On XNLI, FlauBERT reaches 83.4 (large) and 80.6 (base) [arxiv:1912.05372 tab. 5]. The paper's classification head has an extra tanh layer that the authors describe in the text [arxiv:1912.05372 §5.1]; a different head changes the classification numbers, so state which one you use.

Licensing: the paper is CC BY 4.0, but the authors' code repository is licensed for non-commercial use only, and the model weights and the source corpora each need their own check before commercial use [arxiv:1912.05372 §3.1]. Card: [FlauBERT: Unsupervised Language Model Pre-training for French](/papers/1912.05372).

### Monolingual Spanish encoder with whole-word masking (BETO)

BETO is a BERT-Base-sized encoder (12 layers, hidden size 768, 110M parameters) pretrained only on Spanish text, in cased and uncased versions [arxiv:2308.02976 §3] [tab. 1]. The corpus is about 3 billion words from Spanish Wikipedia and OPUS sources such as subtitles, news and government journals [arxiv:2308.02976 §3]. Masking is whole-word and dynamic: each sentence gets 10 different masks, and when one sub-word of a word is masked, the other sub-words of that word are masked too. The masking rate is not stated [arxiv:2308.02976 §3]. Pretraining runs 2M steps in two phases: the first 900k at batch size 2048 and length 128, then the rest at batch size 256 and length 512, with peak learning rate 1e-4 after 10k warm-up steps [arxiv:2308.02976 §3]. Fine-tuning follows BERT, with a grid over batch size, learning rate and epochs [arxiv:2308.02976 §5.1].

Reported effects (author-reported, fine-tuned on each task's Spanish training data): on XNLI the cased model reaches 82.01 against 78.50 for the best multilingual BERT the authors found, and on NER 88.43 against 87.38. Uncased is best on PAWS-X and MLDoc. The paper marks the cased POS score (98.97) and the uncased MLDoc score (96.12) as new state of the art [arxiv:2308.02976 tab. 1]. On question answering the result is mixed: on TAR-to-XQuAD the best multilingual BERT leads on F1 (77.60 against 77.56) [arxiv:2308.02976 tab. 2]. Whether these scores are dev or test is not stated [arxiv:2308.02976 §5.2]. The paper also compiles Spanish tasks into the GLUES suite [arxiv:2308.02976 §4].

Paper vs code: the linked repository holds a README, a model page, a licence and two replication stubs, but no pretraining or fine-tuning code [code: replication/ner-c.md#L1]. Its LICENSE is CC BY 4.0, but the README warns that the authors are not sure all training datasets have licences compatible with CC BY 4.0, "especially for commercial use" [code: README.md#L58-L59]. Check each source before commercial use of the weights or the corpus. Card: [Spanish Pre-trained BERT Model and Evaluation Data](/papers/2308.02976).

### Dutch-only BERT with sentence-order prediction (BERTje)

BERTje is a BERT-base encoder pretrained only on Dutch text, about 12 GB or 2.4 billion tokens from books, news, Wikipedia and a reference corpus [arxiv:1912.09582 §2.1]. It replaces next-sentence prediction with sentence-order prediction, where the second segment is either the next sentence or the previous one [arxiv:1912.09582 §2.2]. The paper describes masking 15% of tokens, masking whole words rather than single pieces [arxiv:1912.09582 §2.2]. Fine-tuning runs for four epochs on each task [arxiv:1912.09582 §4].

Reported effects (author-reported, single runs): on the CoNLL-2002 Dutch NER test set the model scores 88.3 F1, against 80.7 for multilingual BERT [arxiv:1912.09582 tab. 1]. On the 110k Dutch Book Reviews sentiment task it scores 93.0 accuracy, against 89.1 [arxiv:1912.09582 tab. 4]. The authors report it ahead of multilingual BERT on every test column of their POS, semantic-role and spatio-temporal tables as well [arxiv:1912.09582 tab. 2] [tab. 3].

Caveats: the released code is research code, and it does not follow the paper's masking. It masks single wordpieces, and it caps predictions at 20 per sequence, so for full-length inputs the masked fraction is far below 15% [code: pretraining/create-pretrain-data.py#L32] [#L34]. The SoNaR split also differs from the paper [code: finetuning/prepare/prepare-sonar.py#L248]. Two of the five pretraining sources are not named further in the paper, so the corpus cannot be rebuilt from it [arxiv:1912.09582 §2.1]. Card: [BERTje: A Dutch BERT Model](/papers/1912.09582).

### Arabic encoders with a formal-text and a tweet-trained model (ARBERT and MARBERT)

Both models are BERT-Base encoders with a 100K WordPiece vocabulary, and they differ only in data and objective [arxiv:2101.01785 §3.1] [§3.2]. ARBERT is pretrained with whole-word masking on about 61 GB of mostly formal Arabic, and the text gives about 6.5B tokens for it [arxiv:2101.01785 §3.1] [tab. 1]. MARBERT is pretrained on 1B tweets sampled from an in-house collection, with no next-sentence objective, because tweets are short [arxiv:2101.01785 §3.2]. Diacritics are removed, and URLs, user mentions and hashtags become placeholder tokens [arxiv:2101.01785 §3.1] [§3.2].

Reported effects (author-reported, fine-tuned on the test splits of the ARLUE benchmark): MARBERT-v2 scores 77.40, above XLM-R Large at 76.55 and ARBERT at 76.07 [arxiv:2101.01785 §5.2]. The score averages eight table rows rather than the six task clusters named in the abstract, which is our reading of the table [arxiv:2101.01785 §5.2]. The paper never defines "MARBERT-v2" [§5.2]. Caveats: the tweet data is not released [arxiv:2101.01785 §3.2], and the checkpoints are for research use only [code: README.md#L262-L263]. Card: [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](/papers/2101.01785).

### Fine-tuning MARBERT on imbalanced Arabic tweets (sentiment and spam)

This card reuses MARBERT and does no pretraining of its own, so it shows the fine-tuning side of the recipe. MARBERT is a BERT-style Arabic encoder pretrained on dialectal and standard Arabic tweets, about 15.6B tokens, with the next-sentence objective dropped [arxiv:2606.25495 §4.2]. The authors fine-tune it separately for five-class sentiment and for spam, on private customer tweets from a Saudi telecom company [arxiv:2606.25495 §5.1]. Cleaning removes hashtags, usernames and URLs, collapses noise and duplicate tweets, and maps some letter variants to one form [arxiv:2606.25495 §4.1]. The reported setting is Adam with learning rate 2e-5 and epsilon 1e-8, training batch 96, maximum length 256 and 5 epochs, the best of six rows tried [arxiv:2606.25495 tab. 4] [§6]. For the rare classes, the authors rejected over- and undersampling and collected more sarcasm tweets, from about 200 to about 1,400 [arxiv:2606.25495 §5.3].

Reported effects (author-reported, F1): the best sentiment row scores 0.75 [arxiv:2606.25495 tab. 4]. Per class, sarcasm rises from 0.33 at training batch 16 to 0.54 at batch 96, and indeterminate from 0.50 to 0.56 [arxiv:2606.25495 tab. 5]. The spam model reaches 0.98 on more than 24,000 tweets [arxiv:2606.25495 tab. 3].

Caveats: no baseline is run on the private data, and the abstract's claim of accuracy gains rests on literature numbers from other datasets [arxiv:2606.25495 tab. 2]. The spam labels come from a keyword list, so the model may learn that list [arxiv:2606.25495 §5.1]. The data is held by the client and no code is released [arxiv:2606.25495 §5.1]. Card: [Spam and Sentiment Detection in Arabic Tweets Using MARBERT Model](/papers/2606.25495).

### Fine-tuned encoder with one light task head (BERT for E2E-ABSA)

The encoder is BERT-base (12 layers, hidden 768, 110M parameters). A single task layer maps its token states to a BIEOS-style tag per word, which marks aspect spans and their sentiment, and the whole stack is fine-tuned [arxiv:1910.00883 §2, §2.1, §3.3]. The head is one of linear, GRU, self-attention, transformer layer or linear-chain CRF [arxiv:1910.00883 §2.2]. Training uses learning rate 2e-5 for 1500 steps, batch 25 (LAPTOP) or 16 (REST), and checkpoint selection on the dev set from step 1000 onward, every 100 steps, averaged over five seeds [arxiv:1910.00883 §3.1]. The paper does not state the optimizer, schedule, clipping, dropout or maximum sequence length; the released code uses AdamW with a linear schedule and clipping at 1.0 [code: main.py#L171-L177] [code: main.py#L216], and those are the authors' choices rather than paper facts. Freezing BERT gave results the authors call "far from satisfactory" [arxiv:1910.00883 §3.4].

Reported effects (author-reported): a linear head already scores 60.43 F1 on LAPTOP and 73.22 on REST, above the best non-BERT baselines at 60.35 and 72.78 [arxiv:1910.00883 tab. 2]. The richer heads help little: the best are GRU on LAPTOP (61.12) and self-attention on REST (74.72), i.e. up to 0.7 F1 on LAPTOP and 1.5 F1 on REST above the linear head [arxiv:1910.00883 tab. 2]. The REST setting in the released code concatenates released splits, a combination its README calls invalid, so the REST numbers cannot be checked against released data [arxiv:1910.00883 §3.1] [code: README.md#L70-L71].

Card: [Exploiting BERT for End-to-End Aspect-based Sentiment Analysis](/papers/1910.00883).

### Language-specific BERT with a CNN over its last four layers (KUISAIL)

One BERT-base encoder per language, with a convolutional classifier on top. For Arabic the encoder is a BERT the authors trained themselves on about 8.2 billion words [arxiv:2007.13184 §4.3]. The classifier takes the outputs of the last four encoder layers as four channels over a 64-token input. It applies 160 filters in five widths (1 to 5 tokens, 32 each), a ReLU and a global max-pool, then a dense sigmoid output for offensive versus not [arxiv:2007.13184 §4.4] [fig. 1]. All weights are fine-tuned for 10 epochs at learning rate 2e-5, and the epoch with the best dev macro-F1 is kept [arxiv:2007.13184 §4.4]. The paper does not state the loss, dropout or decision threshold [arxiv:2007.13184 §4.4].

Reported effects (author-reported, single runs, no variance given): macro-F1 on the OffensEval 2020 subtask A test sets is 0.897 for Arabic, 0.843 for Greek and 0.814 for Turkish, ranked 4th, 4th and 3rd [arxiv:2007.13184 §1] [arxiv:2007.13184 tab. 2]. The average over the three is 0.851, against 0.841 for language-specific BERT with no CNN. Turkish is 0.002 lower with the CNN [arxiv:2007.13184 tab. 2], a difference computed from the table, not reported by the authors.

Card: [KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media](/papers/2007.13184).

### Open causal decoder with ALiBi (BLOOM)

BLOOM is a 176B-parameter causal decoder-only Transformer trained on ROOTS, a corpus with 46 natural and 13 programming languages [arxiv:2211.05100 §1, §3.1]. Against a vanilla decoder it changes three things: ALiBi attention biases in place of learned or rotary position embeddings, an extra LayerNorm directly after the token embeddings, and tied input and output embeddings [arxiv:2211.05100 §3.2, §3.5 table]. The authors chose the causal objective over encoder-decoder and prefix objectives [arxiv:2211.05100 §3.2]. Shape and optimizer: 70 layers, hidden size 14336, 112 heads, sequence length 2048, global batch 2048 sequences, peak learning rate 6e-5 decaying with a cosine to 6e-6, Adam with (β1, β2) = (0.9, 0.95), weight decay 0.1, gradient clipping at 1.0 and no dropout [arxiv:2211.05100 §3.5 table] [arxiv:2211.05100 §3.5]. Training ran in bfloat16 on 384 A100 80GB GPUs for about 3.5 months [arxiv:2211.05100 §3.4].

Three caveats the card records. The cosine decay never reaches its end, because warmup plus decay tokens exceed the total token budget [arxiv:2211.05100 §3.5]. The paper's ALiBi slope rule (2^(-8i/n)) is not what the reference code computes when the head count is not a power of two, and 112 is not one [arxiv:2211.05100 §3.2] [code: megatron/model/transformer.py#L596-L607]. The Megatron defaults for Adam β2 and dropout differ from the paper's settings, so set them explicitly [code: megatron/arguments.py#L460-L462] [code: megatron/arguments.py#L471].

Reported effects (author-reported, zero- and few-shot prompting): HumanEval pass@1 of 15.52% for the 176B model [arxiv:2211.05100 §4.5]. SuperGLUE results are shown only in a figure, and the paper reports wide spread across prompts [arxiv:2211.05100 §4.2, fig. superglue]. Multitask finetuning on xP3 (BLOOMZ) significantly increases zero-shot performance over pretrained BLOOM [arxiv:2211.05100 §4.7].

Card: [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](/papers/2211.05100).

### Bilingual blank-infilling model at 130B scale (GLM-130B)

GLM-130B is a 130B-parameter English and Chinese model trained on 400B tokens with blank infilling [arxiv:2210.02414 §1]. About 30% of training sequences hold several short spans marked with `[MASK]`, and the rest keep a prefix and mask a tail marked with `[gMASK]` [arxiv:2210.02414 §2.2]. The architecture has 70 layers, hidden size 12,288 and 96 heads, with DeepNorm Post-LN, one-dimensional RoPE and GeGLU [arxiv:2210.02414 §2.1] [app. A]. The stability recipe has three parts: FP16 mixed precision, DeepNorm, and an embedding gradient shrink with factor 0.1 [arxiv:2210.02414 §3]. The authors report that Pre-LN, Sandwich-LN and PB-Relax all failed in their runs [arxiv:2210.02414 §3] [app. A].

Reported effects (author-reported): LAMBADA zero-shot accuracy 80.21 against 76.2 for GPT-3 175B, and 44.75 on 5-shot MMLU against 43.9 [arxiv:2210.02414 §5.1] [§5.2]. Weight-only INT4 quantization with a per-row absolute-maximum scale keeps LAMBADA at 79.47 and MMLU at 44.80, which lets the model run on four 24 GB RTX 3090 cards [arxiv:2210.02414 §4]. Caveats: the training logs record restarts and changes to the shrink factor and learning rate that the one-line summary omits [code: logs/main-log-en.md#L185-L240]; the released weights are for non-commercial research [code: MODEL_LICENSE]. Card: [GLM-130B: An Open Bilingual Pre-trained Model](/papers/2210.02414).

### Scientific corpus with citation and working tokens (Galactica)

Galactica is a decoder-only family from 125M to 120B parameters, trained on a curated 106B-token corpus, of which 83% is papers [arxiv:2211.09085 §4.1] [tab. 2]. Its text is stored as markdown, and special tokens mark citations (`[START_REF]` and `[END_REF]`, with a title and a surname inside), step-by-step working (`<work>`), and SMILES, protein and DNA sequences [arxiv:2211.09085 §3.1] [§3.1.1] [§3.1.2]. Prompt-formatted tasks are included in pre-training instead of added afterwards [arxiv:2211.09085 §3.2]. The models train for 450B tokens, about 4.25 epochs [arxiv:2211.09085 §5.1].

Reported effects (author-reported, zero-shot): LaTeX equation recall of 68.2% for the 120B model, against 49.0% for GPT-3 [arxiv:2211.09085 tab. 6]; 20.4% on MATH with 5-shot mCoT, against 8.8% for PaLM 540B [arxiv:2211.09085 tab. 9]. Caveats: the 120B model was stopped at the start of the fifth epoch, so its final token count is not stated [arxiv:2211.09085 fig. 6]; the authors advise against general-knowledge use [arxiv:2211.09085 §7.1]; the repository gives CC BY-NC 4.0 for the weights [code: LICENSE-MODEL.md]. Card: [Galactica: A Large Language Model for Science](/papers/2211.09085).

### Open multilingual GLM family with tool use (ChatGLM)

The ChatGLM report covers the family from GLM-130B to GLM-4 and GLM-4 All Tools [arxiv:2406.12793 §1]. GLM-4 is pretrained on about ten trillion tokens, mostly Chinese and English [arxiv:2406.12793 §1] [§2]. Against GLM-130B it drops bias terms except for query, key and value, uses RMSNorm and SwiGLU, extends rotary positions to two dimensions, and uses grouped-query attention with a feed-forward width of 10/3 of the hidden size [arxiv:2406.12793 §2]. Context grows from 2K to 128K through position extension, long-text training and long-context alignment [arxiv:2406.12793 §2]. Alignment combines supervised fine-tuning with RLHF [arxiv:2406.12793 §2]. GLM-4 All Tools is trained to pick a browser, a Python interpreter, a text-to-image model or a user-defined function, and to use each tool's result [arxiv:2406.12793 §2].

Reported effects (author-reported): GLM-4 (0520) scores 83.3 on MMLU, 93.3 on GSM8K and 78.5 on HumanEval [arxiv:2406.12793 tab. 2]. On the web, All Tools reaches 91.59 on the GSM8K Python task, against 92.72 for GPT-4 (Web) [arxiv:2406.12793 tab. 9]. The report does not state learning rates, batch sizes, step counts or alignment data sizes [arxiv:2406.12793 §2], so use it to judge relevance rather than to reproduce training. Card: [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](/papers/2406.12793).

### Masked model over clinical code histories (BEHRT)

BEHRT applies a BERT-style encoder to diagnosis histories from CPRD, a UK primary-care database, with 301 diagnosis codes after filtering and about 1.6 million patients [arxiv:1907.09538 §2.2]. Each visit is treated as a sentence of codes. A token's input is the sum of four embeddings: the diagnosis, the patient's age, a segment (visit parity) and the visit position [arxiv:1907.09538 §2.3]. Pre-training is masked disease prediction at a 15% masking rate [arxiv:1907.09538 §2.4]. Fine-tuning adds one output layer per task and predicts several diagnoses at once for the next visit, and within 6 and 12 months [arxiv:1907.09538 §2.4, §2.5]. The paper states the architecture sizes and the masking rate, but not the optimizer, learning rate or sequence length [arxiv:1907.09538 §3, tab. 2].

Reported effects (author-reported): average precision (APS) of 0.462 for the next visit, against 0.360 for DeepR and 0.382 for RETAIN; the same ordering holds at 6 and 12 months [arxiv:1907.09538 tab. 1]. The baselines were tuned with Bayesian optimization, and so were BEHRT's own settings [arxiv:1907.09538 §3, app. A].

Caveat: the data are licensed through CPRD and the vocabulary is not released, so you must build your own code vocabulary before any reuse [arxiv:1907.09538 §2.2]. Card: [BEHRT: Transformer for Electronic Health Records](/papers/1907.09538).

### Table and text encoder over sampled rows (TaBERT)

TaBERT is a BERT-initialized encoder pretrained on text next to tables: 26.6 million examples in total, 1.3 million from English Wikipedia and 25.3 million from the WDC WebTable Corpus [arxiv:2005.08314 §3.2] [app. A.1]. It reads a question together with K rows of the table, not the whole table. Each row becomes one sequence: the question tokens, then one string per cell holding the column name, a data type (`text` or `real`) and the cell value [arxiv:2005.08314 §3.1] [eq. 1]. A BERT model encodes each row separately, and vertical self-attention then links the same position across rows, with three vertical layers at K = 3 [arxiv:2005.08314 §3.1] [§5]. At parse time the rows are the K with the highest n-gram overlap with the question; during pretraining they are sampled at random [arxiv:2005.08314 §3.1] [§5]. Three losses are summed: masked language modelling on the context, masked column prediction, and cell value recovery [arxiv:2005.08314 §3.2] [app. A.2].

Training uses learning rate 2e-5 for K = 1 and 4e-5 for K = 3, weight decay 0.01 and 10 epochs. The largest model took about 6 days on 128 V100 GPUs [arxiv:2005.08314 tab. 6] [app. A.2].

Reported effects (author-reported): on WikiTableQuestions, MAPO with TaBERT-Large at K = 3 averages 51.8 ± 0.6 on test. The run that was best on dev scores 52.3, above the 46.9 of the previous ten-model ensemble [arxiv:2005.08314 tab. 1]. With K = 1, the test mean is 50.4 for TaBERT-Base and 49.4 for BERT-Base [arxiv:2005.08314 tab. 1]. On Spider dev, TaBERT-Large with K = 3 averages 64.5 ± 0.6, against 61.3 ± 1.2 for BERT-Large with K = 1 [arxiv:2005.08314 tab. 2].

Caveats: the released code is non-commercial (CC BY-NC 4.0), and it does not include the two parsers [code: LICENSE.md] [arxiv:2005.08314 app. B]. Its Large K = 3 setup does not run as written, because the hidden size 1024 is not divisible by the default six vertical heads, and the paper gives no head count [code: table_bert/vertical/vertical_attention_table_bert.py#L59-L62]. Pick a head count that divides the hidden size, and state it. Card: [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](/papers/2005.08314).

### Citation-trained document embeddings (SPECTER)

SPECTER starts from SciBERT and continues training it with a triplet margin loss over citation triples: a query paper, a paper it cites and a paper it does not [arxiv:2004.07180 §2.3, eq. 2]. Distance is L2 with margin 1; the authors report that cosine and dot product were worse [arxiv:2004.07180 §2.3]. The input is the title and abstract only, and the output is one vector per paper, used without fine-tuning [arxiv:2004.07180 §2.2, §2.5]. Training uses about 146K query papers and about 684K triples, five per query (two hard, three easy), from a Semantic Scholar subset [arxiv:2004.07180 §4]. The paper states two epochs, learning rate 2e-5 with a slanted triangular schedule, and an effective batch of 32 built from micro-batches of 4 [arxiv:2004.07180 §4]. The authors ran it on one 12 GB GPU, at about one to two days per epoch [arxiv:2004.07180 §4].

Reported effects (author-reported): average 80.0 over the 12 SciDocs metrics, against 76.9 for SGC, the next-best baseline, and 59.6 for SciBERT with no citation training [arxiv:2004.07180 tab. 1].

Caveats: the paper's pooling is the [CLS] vector, but the released code averages tokens, and the maximum length differs between the code paths [arxiv:2004.07180 §2] [code: experiment_configs/simple.jsonnet#L123-L125]. The paper's learning rate is 2e-5, while the released configuration uses 3e-5 [arxiv:2004.07180 §4] [code: experiment_configs/simple.jsonnet#L161]. Fix the pooling and length you use and keep them for training and inference. Card: [SPECTER: Document-level Representation Learning using Citation-informed Transformers](/papers/2004.07180).

### Continued domain pretraining with a tokenizer extension (ChipNeMo)

Start from a base model, not a chat-tuned one. LLaMA2 7B, 13B and 70B are each continued with next-token training on a domain corpus [arxiv:2311.00176 §2.2]. Before that, about 9,000 domain tokens are added to the tokenizer. Each new token's input embedding is the mean of the embeddings of the pieces the old tokenizer produced for it, and its output weights start at zero [arxiv:2311.00176 §2.1, §3.1]. The corpus is mostly proprietary chip-design text, with a public slice of Wikipedia and GitHub code that makes up about 9.2% of training tokens [arxiv:2311.00176 app. Training Data]. Training runs for roughly one epoch of the blend at learning rate 5e-6 with no scheduler, using 256 sequences of 4,096 tokens per step [arxiv:2311.00176 §2.2]. Alignment follows: supervised fine-tuning on 128K general chat samples plus 1.4K domain samples, for one epoch [arxiv:2311.00176 §2.3]. Repeating the domain samples for more than one epoch caused overfitting, and running the same continued pretraining on a chat-aligned model degraded its alignment [arxiv:2311.00176 §2.3].

Reported effects (author-reported, on internal benchmarks): on the design-knowledge multiple-choice set, the 70B model scores 76.6 against 58.4 for GPT-4 and 52.3 for LLaMA2-70B [arxiv:2311.00176 app. Auto Evaluation Results]. On the engineering chatbot with retrieval, it averages 5.68 on a 7-point scale against 4.52 for GPT-4 [arxiv:2311.00176 app. Engineering Assistant Chatbot Human Evaluation]. Full continued pretraining clearly beat LoRA adapters (26.4M and 211.2M trainable parameters) on in-domain tasks [arxiv:2311.00176 §3.4], and the 70B run cost 20,500 A100-hours [arxiv:2311.00176 §3.5]. Retrieval is a separate stage: an e5-small retriever fine-tuned on 3,000 generated samples supplies passages at answer time [arxiv:2311.00176 §2.4]. The domain corpus and the benchmarks are internal, so outsiders cannot rerun them [arxiv:2311.00176 §3]. Card: [ChipNeMo: Domain-Adapted LLMs for Chip Design](/papers/2311.00176).

### Continued pretraining on in-domain tweets (CT-BERT)

CT-BERT starts from BERT-Large (English, uncased, whole-word masking) and keeps training it with masked-token and next-sentence objectives on 22.5M cleaned tweets about COVID-19 [arxiv:2005.07503 §2]. Usernames and URLs are replaced with one shared token each, and emojis become text aliases [arxiv:2005.07503 §2]. The paper reports batches of 1024 sequences of at most 96 tokens, run for 500,000 steps at a constant learning rate of 2e-5 [arxiv:2005.07503 §2, §3.1]. The released code adds a 10,000-step warmup and linear decay instead, so state which schedule you use [code: run_pretrain.py#L267-L270].

Reported effects (author-reported, dev-split F1 after fine-tuning on five classification sets): the mean is 0.833 against 0.802 for BERT-Large [arxiv:2005.07503 tab. 2]. By relative gain, the COVID-19 category set is largest at 25.88% and SemEval is smallest at 8.97% [arxiv:2005.07503 tab. 2]. One of the five sets, SST-2, is not Twitter data and serves as a control [arxiv:2005.07503 §2.1].

Caveats: the paper's printed formula for relative gain has the numerator reversed, and the table values match model minus baseline [arxiv:2005.07503 §3.2]. The paper says 30,000 words for the vocabulary, but the config has 30,522 entries [code: configs/bert_config_covid_twitter_bert.json#L12]. Fine-tuning epochs and learning rate were tuned on BERT-Large and reused for CT-BERT [arxiv:2005.07503 §2.1, §4]. The 22.5M-tweet corpus is not in the repository [arxiv:2005.07503 §5], and the MVS dataset is unpublished [arxiv:2005.07503 §2.1]. Card: [COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter](/papers/2005.07503).

### Continued pretraining of Llama-2 on medical text with replay (MEDITRON)

MEDITRON continues Llama-2 at 7B and 70B on GAP-Replay, a medical mixture of about 46.7B training tokens. It holds clinical guidelines from 16 sources, PubMed abstracts, full texts of about 5M open-access PubMed and PMC papers, and 420M replay tokens from RedPajama, about 1% of the mix, to limit forgetting of general text [arxiv:2311.16079 §2.1] [§2.2] [§2.3] [tab. 1]. Inline citations, figure and table references and formulas are replaced by special tokens [arxiv:2311.16079 §2.2] [app. B.2]. The architecture and tokenizer are Llama-2's, extended with those tokens [arxiv:2311.16079 §4.1] [app. B.2].

Pretraining uses AdamW (β1 0.9, β2 0.95, ε 1e-5), a cosine schedule with 2000 warmup steps and decay to 10% of the peak, weight decay 0.1 and clipping at 1.0 [arxiv:2311.16079 §4.1]. The peak learning rate is 1.5e-4 for 70B and 3e-4 for 7B and 13B, with context 2048 for 7B and 4096 for 70B [arxiv:2311.16079 §4.1]. The 70B run used a global batch of 512, tensor, pipeline and data parallelism of 8, 8 and 2 on 128 A100 80GB GPUs [arxiv:2311.16079 §3] [tab. 1], and took 332 hours [arxiv:2311.16079 app. A]. Each benchmark is then fine-tuned on its own training set for 3 epochs, batch 64, learning rate 2e-5 [arxiv:2311.16079 §4.2]. Three inference modes are compared: the top token, chain-of-thought with the zero-shot cue, and self-consistency over 5 samples at temperature 0.8 with a majority vote [arxiv:2311.16079 §4.3].

Reported effects (author-reported): the 70B model with self-consistency averages 72.0 on the five result columns, against 69.2 for fine-tuned Llama-2-70B in the same mode [arxiv:2311.16079 tab. 5]. In few-shot use without fine-tuning, the 70B average rises from 60.8 at Llama-2 to 63.3 at 23,000 iterations [arxiv:2311.16079 tab. 6].

Caveats: the paper gives the decay target three ways. The text says 10% of peak, while an appendix study used for pretraining ends near 53% of the peak [arxiv:2311.16079 §4.1] [app. D.1] [tab. 11]; the paper states no 70B end rate. The data-mixture ablations use the 7B model only [arxiv:2311.16079 §7.2] [tab. 8]. Only 8 of 16 guideline sources are released [arxiv:2311.16079 §2.1]. The authors say the models are not ready for clinical use [arxiv:2311.16079 app. E]. Card: [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](/papers/2311.16079).

### Continued science-model pretraining on materials text (MatSciBERT)

MatSciBERT does not start from scratch. It starts from SciBERT's uncased weights and keeps training with masked language modeling on about 285M words of materials full text and abstracts, from roughly 154,000 papers [arxiv:2109.15290 §2.1] [tab. 1]. The recipe is RoBERTa-style: dynamic whole-word masking of 15% of words, no next-sentence loss, and sequences of up to 512 tokens that can span several documents [arxiv:2109.15290 §2.2]. It uses AdamW with a peak learning rate of 1e-4 and linear decay, on two V100 GPUs for about 10 days [arxiv:2109.15290 §2.2]. For fine-tuning, the paper compares a softmax head, a CRF and a two-layer BiLSTM-CRF for entity tagging [arxiv:2109.15290 §2.4.1]. Relation classification marks the two entities with special tokens, and abstract classification uses the CLS output [arxiv:2109.15290 §2.4.2] [§2.4.3].

Reported effects (author-reported, mean of 3 seeds, SciBERT as the only baseline): on SOFC-Slot, the BiLSTM-CRF head reaches Macro-F1 65.92 against 61.68 for SciBERT [arxiv:2109.15290 tab. 2]; the text puts the gap at about 4.2 points [§3.3.1]. On glass versus non-glass abstracts, F1 is 93.57 against 91.04 [tab. 6]. On materials-synthesis relation classification the Macro-F1 values are 87.87 and 87.22, and the paper says they are not comparable with prior work [tab. 7] [§3.3.3].

Caveats: the pretraining corpus is not released. Rebuilding it needs publisher full-text access, labelled abstracts and the family classifiers [arxiv:2109.15290 §2.1]. Only the MatSciBERT path normalises its input text, so part of the gain may come from preprocessing [code: ner/ner.py#L54-L62]. The paper describes both a materials vocabulary and SciBERT's vocabulary, and the released code uses SciBERT's [arxiv:2109.15290 §2.2] [§3.2] [code: pretraining/matscibert_pre_train.py#L63]. The repository is GPL-3.0, so only paths and line numbers are cited here. Card: [MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction](/papers/2109.15290).

### Document encoder inside a graph (BertGCN)

The method builds a word-document graph over the whole corpus, in the style of TextGCN. Word-word edges carry positive PMI, and document-word edges carry TF-IDF weights [arxiv:2105.05727 §3.1, eq. 1]. Each document node starts from the BERT or RoBERTa [CLS] embedding, and a two-layer GCN runs over the graph while BERT is trained jointly with it [arxiv:2105.05727 §3.1, §4.1]. Full-batch training would not fit BERT in memory, so a memory bank holds one embedding per document. It is refreshed at the start of each epoch and again for each minibatch, and BERT's learning rate is kept small because the stored rows come from different BERT states [arxiv:2105.05727 §3.3]. The final prediction mixes the GCN output with a BERT-only head, using a weight lambda of 0.7 chosen on a development set [arxiv:2105.05727 §3.2, §4.3, fig. 1]. Author-reported test accuracy with BERT-base is 89.3 on 20NG, 98.1 on R8, 96.6 on R52, 72.8 on Ohsumed and 86.0 on MR. BERT alone scores 85.3, 97.8, 96.4, 70.5 and 85.7 on the same sets [arxiv:2105.05727 tab. 1]. Without the small BERT learning rate, the reported development run did not converge [arxiv:2105.05727 tab. 2]. The released code defaults to RoBERTa and truncates documents at 128 tokens [code: train_bert_gcn.py#L21] [code: train_bert_gcn.py#L25-L26]. The repository has no license, so this page cites it but does not reuse it. Card: [BertGCN: Transductive Text Classification by Combining GCN and BERT](/papers/2105.05727).

### Contrastive pretraining for text and code embeddings (cpt-text, cpt-code)

Paired data takes the place of labels here: neighbouring passages of web text for the text models, and a docstring with its function for the code models [arxiv:2201.10005 §1, §3]. Each side is encoded separately. The embedding is the last-layer state at an end delimiter, and training pulls true pairs together and pushes them away from every other pair in the batch, with a trainable temperature [arxiv:2201.10005 §2.1, §2.2]. The text models start from GPT-family models and the code models from Codex [arxiv:2201.10005 §2.2]. Batch size matters: on a 300M text model, NQ MRR@10 rises from 71.4 at batch 1536 to 84.7 at 12288 [arxiv:2201.10005 tab. 9]. The batch sizes used for the main models shrink as the model grows, and the paper does not give the reason [arxiv:2201.10005 tab. 1].

Reported effects (author-reported): the largest unsupervised text model, with 175B parameters, reaches 91.8 average linear-probe accuracy on seven SentEval tasks, against 88.3 for DECLUTR [arxiv:2201.10005 tab. 2]. On MSMARCO it reaches MRR@10 of 22.7 against 18.4 for BM25 [arxiv:2201.10005 tab. 5]. On CodeSearchNet with 1K candidates, the 1.2B code model averages 93.5 against 77.4 for GraphCodeBERT [arxiv:2201.10005 tab. 7]. Sentence similarity is the weak spot: 71.1 STS average against 79.8 for SimCSE [arxiv:2201.10005 tab. 4]. The paper names its data sources but not their size or filtering, and it has no official code [arxiv:2201.10005 §3]. Card: [Text and Code Embeddings by Contrastive Pre-Training](/papers/2201.10005).

### Continued training against hallucinated facts (FactualityPrompts)

Continued Wikipedia training gets two changes. Each sentence is prefixed with its article title, and the loss ignores tokens before a pivot inside each sentence, by default the middle of the sentence. Decoding stays autoregressive [arxiv:2206.04624 §6.1, §6.2]. At 530B, the vanilla model with top-p decoding has named-entity error of 33.3% on factual prompts and entailment of 11.8%. With both training changes and factual-nucleus decoding, these become 14.5% and 25.5% [arxiv:2206.04624 tab. 5]. Diversity falls from 0.90 to 0.33 in the same comparison [arxiv:2206.04624 tab. 5]. Continued Wikipedia training without the two changes did not help at 1.3B, with named-entity error of 52.5% against 52.4% for the vanilla model [arxiv:2206.04624 tab. 5]. Factual-nucleus decoding shrinks the top-p cut-off as a sentence goes on, resets it at each new sentence and stops it at a floor [arxiv:2206.04624 §5.1]. Most ablations run at 1.3B, and only the best setting is shown at 530B [arxiv:2206.04624 §5.2, §6.3]. The released code covers the metrics and preprocessing, but not the factual-nucleus sampler, and its public checkpoint is smaller than the paper's models [code: README.md#L155]. Card: [Factuality Enhanced Language Models for Open-Ended Text Generation](/papers/2206.04624).

### Pretrained backbone for a web agent (WebShop)

In the shopping environment, a BART model turns the instruction into a search string, and a 12-layer BERT scores each clickable button with cross-attention. Both are trained by imitating recorded human sessions, on 1,421 search pairs and 9,558 choice samples [arxiv:2207.01206 §4.2]. The choice model is then tuned with on-line policy gradient while the search model stays frozen, so that generated queries do not drift from language [arxiv:2207.01206 §4.3]. On 500 held-out instructions, imitation alone scores 59.9 with 29.1% success, and imitation plus RL scores 62.4 with 28.7% success. Human experts score 82.1 with 59.6% success, and the rule baseline scores 45.6 with about 10% [arxiv:2207.01206 tab. 2] [arxiv:2207.01206 §5.2]. Removing language pretraining from the choice model cuts success by nearly two-thirds, a result shown only in a figure [arxiv:2207.01206 §5.2, fig. 4]. RL from raw BERT without imitation starts below the rule baseline [arxiv:2207.01206 §5.2]. Card: [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](/papers/2207.01206).

### Frozen language-pretrained blocks for other data (FPT)

Keep the self-attention and feedforward blocks of a GPT-2 base model, pretrained on natural language, frozen. Train only a new linear input layer, a new linear output layer, the positional embeddings and the layer-norm parameters [arxiv:2103.05247 §2.2]. The authors apply this to bit strings, MNIST and CIFAR-10 patches, ListOps and protein sequences. The layer norms carry much of the effect: on CIFAR-10, training them alone reaches 54%, against 25% for the output layer alone [arxiv:2103.05247 tab. 15]. Freezing them as well drops 12-layer CIFAR-10 from 68.2% to 61.5% [arxiv:2103.05247 tab. 16].

Language pretraining beats a random initialisation under the same freeze: 100% against 75.8% on bit memory, and 68.2% against 61.7% on CIFAR-10 [arxiv:2103.05247 tab. 2]. The authors also report far fewer gradient steps to convergence, about four times fewer on bit memory and forty times fewer on bit XOR [arxiv:2103.05247 tab. 6].

Reported effects (author-reported): MNIST reaches 98.0% against 99.1% for a fully trained transformer. CIFAR-10 reaches 72.1% with a 36-layer model, against 70.3% for a fully trained 3-layer GPT-2 [arxiv:2103.05247 tab. 1]. Frozen random LSTMs do much worse, for example 16.8% on ListOps against 34.3% for a frozen random transformer [arxiv:2103.05247 tab. 3] [arxiv:2103.05247 tab. 4].

Caveats: the released code uses an embedding table, not a linear map, for ListOps and homology, reads the bit outputs from the last 20 tokens, clips gradients, and trains every parameter whose name contains "ln" or "norm" [arxiv:2103.05247 §2.2] [code: universal_computation/experiment.py#L60] [code: universal_computation/fpt.py#L184-L192] [code: universal_computation/fpt.py#L132-L150]. Card: [Pretrained Transformers as Universal Computation Engines](/papers/2103.05247).

### Multi-task text-to-text pretraining for dialogue (PPTOD)

One T5 model handles every sub-task of a task-oriented dialogue system: intent classification, belief-state tracking, dialogue action prediction and response generation. Each sub-task is a text-to-text sample made of a short task prompt, the earlier turns, and the expected output text [arxiv:2109.14739 §3.2]. Pretraining uses eleven human-written dialogue corpora with over 2.3M utterances. Each corpus is annotated for only some sub-tasks, and a sample is made only for the annotations it has [arxiv:2109.14739 §3.1] [arxiv:2109.14739 tab. 1]. Pretraining maximises the likelihood of the target text, and fine-tuning on a target task uses the same objective [arxiv:2109.14739 §3.2] [arxiv:2109.14739 §3.3].

For end-to-end use, the model predicts the belief state, queries the database with it, and then generates the action and the response from the same database result [arxiv:2109.14739 §4.1.1]. The authors call this a plug-and-play formulation.

Reported effects (author-reported): PPTOD-base has the top MultiWOZ 2.0 combined score in the paper's table, 102.92 [arxiv:2109.14739 tab. 2]. With 1% of MultiWOZ 2.0 training data, PPTOD-large reaches joint goal accuracy 31.46, against 13.21 for SOLOIST [arxiv:2109.14739 tab. 5]. The annotation mix matters most at 1%: with no annotation types, end-to-end success is 36.08; with all four, 50.90 [arxiv:2109.14739 tab. 8]. A T5-small fine-tuned without multi-task pretraining, with plug-and-play generation and no database state, scores 97.27 combined at 14.17 ms per example. SOLOIST scores 95.74 at 208.69 ms [arxiv:2109.14739 tab. 6]. At 10 samples per intent, intent classification is not the best: USE+ConveRT reaches 85.19 against 84.12 for PPTOD-large [arxiv:2109.14739 tab. 7].

Caveats: the paper's loss is a sum over target tokens, but the released code averages over the non-padding target tokens [arxiv:2109.14739 §3.2] [code: Pretraining/dataclass.py#L290]. The paper states no fine-tuning optimizer, and the released fine-tuning code uses Adafactor at a much higher learning rate than the Adam setting for pretraining [code: E2E_TOD/learn.py#L63]. Card: [Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System](/papers/2109.14739).

### Long-input text-to-text encoder with transient global attention (LongT5)

LongT5 keeps the T5 encoder-decoder and its text-to-text format, and changes only the encoder's self-attention. Each token attends to a sliding window of radius r = 127 on each side, which adds no parameters [arxiv:2112.07916 §3.1.1]. The TGlobal variant cuts the input into blocks of k = 16 tokens and makes one global token per block, from the summed block embeddings and a normalisation. These globals are rebuilt in every layer and not stored, so no side inputs need designing for each task [arxiv:2112.07916 §3.1.2]. Its state is small: the paper counts about 10k extra parameters for base, 25k for large and 50k for xl [arxiv:2112.07916 §3.1.2, fn. 1]. Earlier long-input models (ETC, BigBird, Longformer) needed such side inputs and used masked-language pretraining [arxiv:2112.07916 §1, §6]. The decoder is unchanged, because the targets are short [arxiv:2112.07916 §3.1].

Pretraining on C4 uses principle-sentence generation from PEGASUS: sentences are scored by ROUGE-F1 against the rest of the document, the top ones are masked, and the model writes them out as one target. The masked-sentence ratio is 0.2, not PEGASUS's 0.45 [arxiv:2112.07916 §3.2] [§4.1.1]. In the authors' ablation on NQ and arXiv, this objective beats span corruption, and using both objectives is worse than using this one alone [arxiv:2112.07916 tab. 5] [§5.3].

Reported effects (author-reported, one run per configuration): on PubMed, xl at 16k reaches ROUGE-1 50.23 against 48.36 for HAT-BART, and on BigPatent 76.87 against 60.64 for BigBird-PEGASUS (large) [arxiv:2112.07916 tab. 2]. On TriviaQA, xl at 16k reaches F1 84.83, above BigBird-ETC's 84.5 [arxiv:2112.07916 tab. 4]. Multi-News goes to PRIMER and CNN/DM to HAT-BART [arxiv:2112.07916 tab. 2].

Cost and limits: at input length 2048, LongT5 base TGlobal trains at 765 sequences per second, against 479 for T5.1.1 base, and Local base at 860 [arxiv:2112.07916 §5.1]. The longest inputs that fit on the same TPU slice are 6k for T5.1.1 base, 12k for TGlobal base and 36k for Local base [arxiv:2112.07916 §5.1]. Local attention is faster but trails TGlobal and T5.1.1 on NQ [arxiv:2112.07916 §5.2]. Decoding is greedy, and the decoder's full attention means long outputs are not addressed [arxiv:2112.07916 §4.1] [§3.1] [§6]. Caveats: the paper's comparisons use different input lengths per dataset [arxiv:2112.07916 tab. 2]; one T5.1.1 base entry at 6k has identical EM and F1 (56.73) and may be a transcription error [arxiv:2112.07916 tab. 4]. The released code has the PEGASUS task and data pipeline, but the architecture lives in Flaxformer, which is not pinned to a commit [code: README.md#L79]. Card: [LongT5: Efficient Text-To-Text Transformer for Long Sequences](/papers/2112.07916).

### Gated attention with exact attention inside chunks and linear attention across them (FLASH)

A single softmax-free attention head, squared-ReLU weighted, is gated by a GLU-style unit, so the attention itself can be weak [arxiv:2202.10447 §2, eq. 3, eq. 5]. The sequence is cut into fixed-size chunks. Each chunk runs exact attention on itself, and a per-chunk summary carries information across chunks, so cost stays linear in length for a fixed chunk size [arxiv:2202.10447 §3.2]. For causal language modeling the cross-chunk term sums only earlier chunks, so the sequential loop runs over chunks rather than tokens [arxiv:2202.10447 §3.2, §3.2.1]. Pretraining uses 2^18 tokens per batch, 125K steps and a peak learning rate of 7e-4 with AdamW in the reported runs [arxiv:2202.10447 tab. 6] [arxiv:2202.10447 tab. 7].

Reported effects (author-reported): on Wiki-40B language modeling, the speedup over Transformer++ to reach its quality is 1.2x to 4.9x across contexts from 512 to 8192 [arxiv:2202.10447 §4.2]; on C4 masked-LM it is 1.0x to 4.8x [arxiv:2202.10447 §4.1]. At context 8192 on PG-19, perplexity is 41.07 against 43.27 for Transformer+, with a 12.12x speedup against that baseline [arxiv:2202.10447 tab. 3]. These speedups are training-cost ratios, not per-step latency [arxiv:2202.10447 §4.2]. No official code is known; the appendix listings disagree with the equations on scaling and masking, so implementers should test both variants [arxiv:2202.10447 app. D].

Card: [Transformer Quality in Linear Time](/papers/2202.10447).

### Multilingual causal LM for few-shot use (mGPT)

Two GPT-3-style decoder-only models, about 1.3B and 13B parameters, are trained with next-token prediction on Wikipedia and C4 in 61 languages [arxiv:2204.07580 §3.1] [arxiv:2204.07580 §3.3] [arxiv:2204.07580 tab. 1]. Documents are deduplicated by a 64-bit hash. C4 is filtered, and a Vowpal Wabbit classifier separates Wikipedia documents from the filtered C4 documents as a quality signal [arxiv:2204.07580 §3.1].

The tokenizer is byte-level BPE with the default strategy, chosen by held-out perplexity over five variants [arxiv:2204.07580 §3.2] [arxiv:2204.07580 tab. 3]. Training uses a 512-token context, a total batch of 2048 and 600k steps, with 400B tokens stated [arxiv:2204.07580 §3.4]. The 1.3B model took 14 days on 256 V100 GPUs and the 13B model 22 days on 512 V100 GPUs [arxiv:2204.07580 §3.4].

Classification scores each label by the summed negative log-likelihood of its completed prompt, and takes the lowest [arxiv:2204.07580 §4.2.1]. NER and POS tag one word at a time, filling each candidate tag into the prompt [arxiv:2204.07580 §4.2.2].

Reported effects (author-reported): mGPT-1.3B is close to XGLM-1.7B on classification, with fewer weights and twice the languages [arxiv:2204.07580 §4.2.1] [arxiv:2204.07580 tab. 7]. mGPT-13B beats XGLM-7.5B zero-shot on four of the five classification tasks. In few-shot settings it trails on XWINO and XCOPA at one and four shots, and leads on PAWSX at every shot count and on XNLI at one and four shots [arxiv:2204.07580 tab. 7]. Hate-speech accuracy stays near the 50% chance level [arxiv:2204.07580 tab. 8]. Its 4-shot NER and POS scores are far below fine-tuned encoders such as XLM-R, which the authors note is not a like-for-like comparison [arxiv:2204.07580 tab. 9] [arxiv:2204.07580 tab. 10] [arxiv:2204.07580 §4.2.2].

Caveats: the released code covers evaluation and inference only. The learning rate, optimizer, schedule, head count and final vocabulary are not stated [arxiv:2204.07580 §3.4]. The tokenizer ablation used a 2048-token context, and the final models 512 [arxiv:2204.07580 §3.2] [arxiv:2204.07580 §3.4]. The stated 400B tokens do not match 2048 sequences of 512 tokens over 600k steps, and the paper does not explain the gap. Perplexity is normalised by characters in §3.2 and by tokens in §4.1 [arxiv:2204.07580 eq. 1] [arxiv:2204.07580 §4.1]. The code licence is unclear, since the README says MIT and the LICENSE file says otherwise [code: LICENSE.txt#L1-L3]. Card: [mGPT: Few-Shot Learners Go Multilingual](/papers/2204.07580).

### Token-free byte-level encoder with a learned tokenizer (UTC)

UTC reads UTF-8 bytes and needs no vocabulary. A learnable tokenizer adapted from Charformer scores blocks of one to four bytes at each position, mixes them with a softmax, and mean-pools the result to half the byte length [arxiv:2202.11176 §3.1] [§4.2]. A T5-style encoder-decoder with relative attention biases processes those latent subwords [arxiv:2202.11176 §3.2] [app. A]. Pretraining is span denoising on byte spans averaging 20 bytes, covering 15% of the input, with an equal mix of a proprietary Perspective corpus and mC4 [arxiv:2202.11176 §3.4]. For classification the decoder is dropped, the encoder output goes through a GeLU layer, the first position is pooled, and a linear layer with sigmoid cross-entropy gives one probability per label [arxiv:2202.11176 §3.3].

Reported effects (author-reported, proprietary Production-Multilingual set): average AUC-ROC 0.910 for the 102M model and 0.919 for the 268M model, against 0.840 for the in-house multilingual BERT and 0.906 for mT5-base [arxiv:2202.11176 tab. 1]. On the public JMTCC benchmark the 268M model reaches 0.9367, against 0.9239 for mT5-base [arxiv:2202.11176 tab. 2]. On code-switched text the 102M model is below mT5-base (0.9191 against 0.9393), while the 268M model is best [arxiv:2202.11176 tab. 3].

Caveats: no official code is linked, and the proprietary pretraining corpus means pretraining cannot be repeated exactly [arxiv:2202.11176 §3.4]. The pretraining loss as printed is ambiguous, and the number of labels per task is not stated [arxiv:2202.11176 §3.3]. Card: [A New Generation of Perspective API: Efficient Multilingual Character-level Transformers](/papers/2202.11176).

### Text and knowledge graph in one encoder (DRAGON)

One encoder reads a text segment and a small local knowledge-graph subgraph at once. The two streams exchange information in both directions after each fusion layer [arxiv:2210.09338 §2.2] [arxiv:2210.09338 fig. 1]. The subgraph is built by linking mentions to graph nodes, adding two-hop bridge nodes, and capping the graph at 200 nodes [arxiv:2210.09338 app. B.1]. The encoder has 19 unimodal language-model layers and 5 fusion layers, about 360M parameters in total [arxiv:2210.09338 §3.1] [arxiv:2210.09338 tab. 7].

Pretraining sums two losses with equal weight: masked language modeling on the text, and link prediction on held-out graph edges scored with DistMult [arxiv:2210.09338 §2.3] [arxiv:2210.09338 tab. 5]. The general model pretrains on BookCorpus with ConceptNet, and the biomedical one on PubMed with UMLS [arxiv:2210.09338 §3.1] [arxiv:2210.09338 §4].

Reported effects (author-reported): on nine commonsense QA tasks, DRAGON is the best system in every column of [arxiv:2210.09338 tab. 1]. A simple average of its nine columns is about 4.4 points above the continued-MLM RoBERTa baseline. The paper states no average, so this figure is computed for this page [arxiv:2210.09338 tab. 1]. On MedQA it scores 47.5, against 44.6 for BioLinkBERT [arxiv:2210.09338 tab. 6]. In the ablation, MLM alone gives 74.3 on CSQA and link prediction alone 73.8, against 76.0 for both [arxiv:2210.09338 tab. 5].

Caveats: the link loss as printed has a negative-term sign that leaves it without a finite minimum. The released code uses a bounded form instead [arxiv:2210.09338 §2.3] [arxiv:2210.09338 eq. 10] [code: modeling/modeling_gnn.py#L216]. The released pretraining script is a CommonsenseQA demo, not the recipe the paper describes [code: scripts/run_pretrain.sh#L9]. The authors name generation as future work, since the model encodes and scores but does not write text [arxiv:2210.09338 §5]. Card: [Deep Bidirectional Language-Knowledge Graph Pretraining](/papers/2210.09338).

### Open decoder on public data (LLaMA)

Decoder-only Transformers from 7B to 65B parameters (the paper's table labels them 6.7B to 65.2B). The two smaller models see 1.0T tokens and the two larger ones 1.4T [arxiv:2302.13971 tab. 2] [arxiv:2302.13971 fig. 1]. Each block normalises its inputs with pre-norm RMSNorm, uses SwiGLU feed-forward layers and rotary embeddings, and has no absolute position embeddings [arxiv:2302.13971 §2.2]. The data is public: English CommonCrawl 67%, C4 15%, GitHub 4.5%, Wikipedia 4.5%, Books 4.5%, arXiv 2.5% and Stack Exchange 2%. GitHub is limited to Apache, BSD and MIT projects [arxiv:2302.13971 §2.1]. Training uses AdamW with betas (0.9, 0.95), weight decay 0.1, gradient clipping at 1.0, 2,000 warmup steps and a cosine decay to 10% of the peak rate [arxiv:2302.13971 §2.3]. The reported throughput of about 380 tokens per second per GPU on 2,048 A100-80GB GPUs works out to roughly 21 days for 1.4T tokens [arxiv:2302.13971 §2.4]. The paper does not state dropout, bias terms or weight tying [arxiv:2302.13971 §2.2].

Reported effects (author-reported): LLaMA-13B beats GPT-3 (175B) on most common-sense benchmarks, and LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B [arxiv:2302.13971 §3.1] [arxiv:2302.13971 tab. 3]. The released code covers inference only [arxiv:2302.13971 §2.2].

Card: [LLaMA: Open and Efficient Foundation Language Models](/papers/2302.13971).

### Open bilingual decoder with chat alignment and tool use (Qwen)

Decoder-only Transformers in three sizes, 1.8B, 7B and 14B parameters, pretrained on up to 3T tokens. The data mixes public web text, encyclopedias, books and code, with a large English and Chinese share [arxiv:2309.16609 §2.1] [arxiv:2309.16609 tab. 1]. Web text is extracted from HTML, language-identified, deduplicated by exact match and by MinHash with LSH, then filtered with rules and model-based quality scorers [arxiv:2309.16609 §2.1]. The tokenizer is byte-pair encoding starting from `cl100k_base`, extended with common Chinese characters and words, with a vocabulary of about 152K [arxiv:2309.16609 §2.2]. The architecture uses rotary embeddings, pre-norm RMSNorm and SwiGLU feed-forward layers with width 8/3 of the hidden size; biases appear only in the query, key and value projections, and the input embedding and output projection are untied [arxiv:2309.16609 §2.3]. Pretraining uses AdamW (β₁ 0.9, β₂ 0.95), context 2048, cosine decay to 10% of the peak rate and BFloat16 mixed precision [arxiv:2309.16609 §2.4].

Long inputs can be handled without training. The paper describes NTK-aware base scaling, dynamic NTK scaling, LogN attention scaling and per-layer window attention, all in words only; it gives no formulas and does not state the window sizes [arxiv:2309.16609 §2.5]. Author-reported perplexity on arXiv papers stays low at 8192 and 16384 tokens with all techniques on [arxiv:2309.16609 tab. 3].

Alignment has three stages. Supervised fine-tuning uses chat conversations in ChatML format, with system and user turns masked from the loss [arxiv:2309.16609 §3.1.1] [§3.1.2]. A reward model is initialised from a same-size model and trained on human comparisons with a pooled end-token reward [arxiv:2309.16609 §3.2.1]. PPO then runs four models at once (policy, value, reference and reward), samples two responses per query, and mixes in a pretraining-data gradient so that benchmark results outside code and maths are not hurt [arxiv:2309.16609 §3.2.2]. Tool use is trained as about 2,000 self-instructed samples in the general SFT mix, using ReAct-style prompts [arxiv:2309.16609 §3.4]. Two specialised models follow: Code-Qwen continues pretraining on about 90B code tokens [arxiv:2309.16609 §4.1], and Math-Qwen-Chat is fine-tuned directly from the base model on a maths instruction set [arxiv:2309.16609 §5.1].

Reported effects (author-reported): the 14B base scores 66.3 on MMLU and 72.1 on C-Eval, both 5-shot [arxiv:2309.16609 tab. 2]. The 14B reward model reaches 74.78 pairwise test accuracy on Helpful-base, against 62.68 for the preference-model baseline [arxiv:2309.16609 tab. 4]. The 14B chat model scores 66.5 on MMLU against 83.0 for GPT-4, and the paper says the RLHF model beats the SFT models and trails GPT-4 [arxiv:2309.16609 tab. 5] [§3.3].

Card: [Qwen Technical Report](/papers/2309.16609).

### Open decoder with a published training logbook (OPT)

OPT is a suite of decoder-only Transformers from 125M to 175B parameters, trained with a GPT-3-style recipe: AdamW with betas (0.9, 0.95), weight decay 0.1, dropout 0.1, and a learning rate that warms up and then decays linearly to 10% of its peak over 300B tokens [arxiv:2205.01068 §2.2] [arxiv:2205.01068 tab. 1]. The batch is held constant at 0.5M to 4M tokens depending on size [arxiv:2205.01068 tab. 1]. The corpus is about 180B tokens: RoBERTa's data, a subset of the Pile, and Pushshift Reddit threads, with only the longest comment chain kept from each thread and MinHash deduplication [arxiv:2205.01068 §2.3]. The 175B model ran on 992 A100 80GB GPUs with FSDP and Megatron tensor parallelism [arxiv:2205.01068 §2.4].

Training was unstable. The paper records at least 35 manual restarts and more than 100 hosts cycled out over two months. When the loss diverged, the team lowered the learning rate and restarted from an earlier checkpoint [arxiv:2205.01068 §2.5]. The released launcher uses a lower peak learning rate for 175B than the table in the paper, so the card records both values [arxiv:2205.01068 tab. 1].

Reported effects (author-reported): OPT-175B is comparable to GPT-3 on the average of standard zero- and few-shot tasks [arxiv:2205.01068 §3.1] [arxiv:2205.01068 fig. 3]. It does worse than GPT-3 on toxicity and CrowS-Pairs [arxiv:2205.01068 §4.2] [arxiv:2205.01068 §4.4]. The stated carbon footprint is 75 tCO2eq, against an estimated 500 tons for GPT-3 [arxiv:2205.01068 §6].

Card: [OPT: Open Pre-trained Transformer Language Models](/papers/2205.01068).

### Sparse mixture-of-experts decoder (GLaM)

GLaM replaces the feed-forward sub-layer of every other block in a decoder-only Transformer with a mixture of 64 experts. Each token is routed to its two highest-scoring experts, and the output is their weighted sum [arxiv:2112.06905 §4] [arxiv:2112.06905 fig. 2]. The gate is a softmax over experts. Whether the two weights are renormalised is not stated [arxiv:2112.06905 §4]. The training loss is next-token cross-entropy plus a GShard-style load-balancing term with weight 0.01 [arxiv:2112.06905 §5.2]. The learning rate is constant for the first 10K steps and then decays with the inverse square root of the step count [arxiv:2112.06905 §5.2]. The weights and computation of the large models are sharded with GSPMD on TPU-v4 [arxiv:2112.06905 §4] [arxiv:2112.06905 app. C].

Scale: the 64B/64E model has 1.2T total parameters and 96.6B active per token [arxiv:2112.06905 tab. 4]. Its corpus is 1.6T tokens, with web pages filtered by a classifier that favours curated text and then mixed with books, Wikipedia, forums, news and conversation [arxiv:2112.06905 §3] [arxiv:2112.06905 tab. 3].

Reported effects (author-reported): GLaM beats GPT-3 (175B) on the average of 29 zero-, one- and few-shot tasks, with about half of GPT-3's inference FLOPs per token and about a third of its training energy [arxiv:2112.06905 tab. 1].

Card: [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](/papers/2112.06905).

### Retrieval-enhanced pretraining (RETRO)

RETRO predicts text in 64-token chunks. For each chunk it retrieves neighbour passages from a database of up to about 2 trillion tokens. The keys are time-averaged frozen BERT embeddings of the neighbour text [arxiv:2112.04426 §2.2] [arxiv:2112.04426 §2.3] [arxiv:2112.04426 §4.1]. Neighbours from the training document itself are removed, and neighbours for the training data are fetched in advance [arxiv:2112.04426 §2.3].

Retrieved passages are encoded by a two-layer bidirectional encoder of width 896 that adds about 19M parameters. They enter through chunked cross-attention in some decoder layers. The first chunk does not depend on retrieval [arxiv:2112.04426 §2.2] [arxiv:2112.04426 §2.4] [arxiv:2112.04426 §2.5] [arxiv:2112.04426 alg. 1]. At sampling time, retrieval runs again after each chunk, so its cost grows with the number of chunks [arxiv:2112.04426 §2.4].

RETRO-fitting adds the new retrieval weights to a frozen pretrained baseline and trains only those. Without retrieval, the frozen model's output is unchanged [arxiv:2112.04426 §4.2] [arxiv:2112.04426 app. B.5]. Reported effects (author-reported): the 7.5B model outperforms Jurassic-1 and Gopher on a majority of Pile test sets [arxiv:2112.04426 §4.1] [arxiv:2112.04426 app. F.1]. The authors' hyperparameters are partly inconsistent with each other, and no official code was found [arxiv:2112.04426 app. C].

Card: [Improving language models by retrieving from trillions of tokens](/papers/2112.04426).

### Linked-document pairs for masked pretraining (LinkBERT)

LinkBERT keeps BERT's masked-word objective but changes what one training input contains. An input holds an anchor segment and a second segment. The second segment is the next segment of the same document, a segment from an unrelated document, or a segment from a document the anchor links to, each with probability one third in the main setting [arxiv:2203.15827 §4.2, §5.1]. Linked documents are drawn with probability inversely proportional to their in-degree, so heavily linked pages do not dominate [arxiv:2203.15827 §4.3]. A three-way relation head on the first token replaces BERT's next-sentence prediction, and the loss is the masked-word loss plus the relation loss with equal weight [arxiv:2203.15827 §4.2] [eq. 1]. The tiny model starts from random weights; base and large continue from BERT checkpoints [arxiv:2203.15827 §5.1].

Reported effects (author-reported): average MRQA F1 is 54.1 for LinkBERT-tiny against 50.0 for BERT-tiny, 77.8 against 75.2 for base, and 81.0 against 78.5 for large [arxiv:2203.15827 tab. 1]. The largest gains are on multi-document sets such as HotpotQA, where base rises from 76.0 to 78.2 [arxiv:2203.15827 tab. 1] [§5.3]. With 10% of the fine-tuning data, base scores 70.5 on HotpotQA against 64.8 for BERT-base [arxiv:2203.15827 tab. 4]. In the biomedical setting, BioLinkBERT-base reaches a BLURB score of 83.39, against 81.10 for PubMedBERT-base at the same size [arxiv:2203.15827 tab. 7]. On tiny, random links match BERT-tiny, and TF-IDF neighbours do better than random but not as well as hyperlinks [arxiv:2203.15827 §5.5] [tab. 5]. Removing the relation loss from base costs 1.7 F1 on HotpotQA and 0.8 on SQuAD [arxiv:2203.15827 tab. 6].

Caveats: the paper does not state the segment length in link-based inputs, how a linked segment is chosen, or what happens to an anchor with no outgoing links [arxiv:2203.15827 §4.2, §4.3]. The released repository holds fine-tuning and preprocessing scripts and models, but not the pretraining code [code: README.md#L24] [code: README.md#L30-L33]. The MRQA split is a halved dev set, so its numbers are not comparable to the official leaderboard [arxiv:2203.15827 §5.2]. Card: [LinkBERT: Pretraining Language Models with Document Links](/papers/2203.15827).

### Fully documented training suite (Pythia)

Pythia is eight decoder-only models from 70M to 12B parameters. Each is trained on the Pile and on a near-duplicate-removed Pile of about 207B tokens, with the same data order [arxiv:2304.01373 §2.2] [arxiv:2304.01373 §2.4]. The architecture is dense in every layer, uses rotary embeddings, runs attention and the feed-forward sub-layer in parallel, and has untied input and output embeddings [arxiv:2304.01373 §2.3]. Every size uses a global batch of 1024 sequences of 2048 tokens and 143,000 steps, about 300B tokens in total [arxiv:2304.01373 §2.4] [arxiv:2304.01373 app. B] [arxiv:2304.01373 app. E]. The optimizer is Adam with betas (0.9, 0.95) and epsilon 1e-8, and the learning rate decays with a cosine to 0.1 times the peak [arxiv:2304.01373 app. E] [arxiv:2304.01373 app. B].

The models, checkpoints, training code and data-loader tools are released [arxiv:2304.01373 §2.7]. Use the suite when you need a public model whose training history you can reproduce. Each model has 154 saved checkpoints [arxiv:2304.01373 §1].

Card: [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](/papers/2304.01373).

### German single-language RoBERTa (GottBERT)

GottBERT is a RoBERTa-base or RoBERTa-large encoder trained only on German text. The authors state that it is the first German single-language RoBERTa model [arxiv:2012.02110 §1] [arxiv:2012.02110 §6]. There are two corpora: the German part of OSCAR (145GB), and a 121GB version filtered with rules and a one-class SVM [arxiv:2012.02110 §3]. Base and large are each trained on both corpora, which gives four models, and both the best and last checkpoints are kept [arxiv:2012.02110 §3] [arxiv:2012.02110 §4].

Pretraining runs 100k updates with batches of 8k sequences, with a peak learning rate of 4e-4 for base and 1.5e-4 for large [arxiv:2012.02110 §3]. Reported effects (author-reported): base models lead on 4 of 6 tasks, and the best filtered and unfiltered checkpoints differ by at most 0.85 points. No seed variance or significance test is reported [arxiv:2012.02110 tab. 3].

Card: [GottBERT: a pure German Language Model](/papers/2012.02110).

### Compute-optimal multilingual model (PaLM 2)

PaLM 2 is a Transformer in three sizes (S, M and L). Its sizes and architecture are withheld [arxiv:2305.10403 app. D]. The report states three ingredients: compute-optimal scaling, in which data and parameters grow at about the same rate [arxiv:2305.10403 §1] [arxiv:2305.10403 §2.1]; a more multilingual and deduplicated pretraining mixture [arxiv:2305.10403 §3]; and a tuned mixture of objectives motivated by UL2 [arxiv:2305.10403 §1]. The scaling study trained at 1e19 to 1e22 FLOPs and chose token counts with FLOPs ≈ 6ND. It fitted a quadratic at each budget and then a power law across budgets [arxiv:2305.10403 §2.1] [arxiv:2305.10403 app. A.1]. Its scaling models (400M to 15B) are not PaLM 2 models [arxiv:2305.10403 §2.1]. The earlier PaLM's non-code data was about 78% English [arxiv:2305.10403 §1].

Two safety-related pretraining choices: a small fraction of documents carry a toxicity control token from a fixed Perspective API version, and the model can be conditioned on low, medium or high toxicity at inference [arxiv:2305.10403 §3] [arxiv:2305.10403 §5.1]. Synthetic canary sequences measure memorisation [arxiv:2305.10403 §4]. Reported effects (author-reported): BIG-Bench Hard 78.1 with chain-of-thought, against 65.2 for PaLM [arxiv:2305.10403 tab. 6], and GSM8K 91.0 with self-consistency, against 74.4 for PaLM [arxiv:2305.10403 tab. 7]. Learning rate, batch size and token count are not stated, so the card treats it as a results-and-recipe reference.

Card: [PaLM 2 Technical Report](/papers/2305.10403).

### Small open model with a curated data mix (Gemma)

Gemma 2B and 7B are open decoder-only Transformers, pretrained on 3T and 6T tokens respectively, with instruction-tuned and RLHF-tuned variants [arxiv:2403.08295 §Pretraining]. The authors say the models use the same ideas and training recipes as their larger Gemini models, scaled down [arxiv:2403.08295 §Pretraining]. The technical report gives architecture sizes and benchmark results, but no learning rate or batch size [arxiv:2403.08295 §Training Infrastructure]. Reported effects (author-reported): the 7B pretrained model averages 56.9 over 18 academic benchmarks, against 54.5 for Mistral 7B [arxiv:2403.08295 tab. 6].

Card: [Gemma: Open Models Based on Gemini Research and Technology](/papers/2403.08295).

### Data-quality-first small model (Phi-3)

phi-3-mini has 3.8B parameters and is trained on 3.3T tokens. Its layout is Llama-2-style, with a 4K default context [arxiv:2404.14219 §1] [arxiv:2404.14219 §2] [arxiv:2404.14219 tab. 3]. Pretraining has two sequential phases: web data for general knowledge, then more heavily filtered web data mixed with synthetic reasoning data [arxiv:2404.14219 §2]. Post-training is supervised fine-tuning followed by DPO [arxiv:2404.14219 §2]. Reported effects (author-reported): results close to Mixtral 8x7B and GPT-3.5 [arxiv:2404.14219 tab. 3]. A 4-bit build runs offline on an iPhone 14 at more than 12 tokens per second [arxiv:2404.14219 §2] [arxiv:2404.14219 fig. 2]. No official code was found, and the report leaves core training hyperparameters unstated [arxiv:2404.14219 §2].

Card: [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](/papers/2404.14219).

### Continued code pretraining with infilling (Code Llama)

Code Llama starts from Llama 2 weights and tokenizer and continues next-token training on code: 500B tokens for the 7B, 13B and 34B models, and 1T for 70B [arxiv:2308.12950 §2.1] [arxiv:2308.12950 §2.6]. The mix is 85% code, 8% natural language related to code and 7% general natural language [arxiv:2308.12950 tab. 1] [arxiv:2308.12950 §2.2]. The Python variant adds 100B tokens, 75% of them Python [arxiv:2308.12950 tab. 1].

Infilling: documents are split into prefix, middle and suffix, then reordered in prefix-suffix-middle or suffix-prefix-middle order, with four special tokens [arxiv:2308.12950 §2.3]. On average it costs 0.6 points at 7B and 1.1 at 13B across HumanEval and MBPP [arxiv:2308.12950 §3.2]. Long-context fine-tuning uses 16,384-token sequences, raises the RoPE base from 10^4 to 10^6 and uses a learning rate of 2e-5 [arxiv:2308.12950 §2.4] [arxiv:2308.12950 §2.6]. Only the 70B base model received it [arxiv:2308.12950 §2.1]. Instruction tuning uses about 5B tokens, including about 14,000 self-generated question, test and solution triples checked by unit tests [arxiv:2308.12950 §2.5]. Reported effects (author-reported): HumanEval pass@1 of 33.5 to 53.0 for the base models, and up to 67.8 for Instruct 70B [arxiv:2308.12950 tab. 2].

Card: [Code Llama: Open Foundation Models for Code](/papers/2308.12950).

### Small code model on filtered and synthetic textbook data (phi-1)

phi-1 is a 1.3B decoder-only Transformer for Python, trained in two stages: pretraining on about 7B tokens of filtered code and synthetic textbooks, then fine-tuning on under 200M tokens of exercises [arxiv:2306.11644 §2] [tab. 1]. The filtered code is the Python part of The Stack and StackOverflow. A random forest, trained on GPT-4 ratings of about 100k samples with CodeGen-Mono embeddings as features, scores the pool. The score threshold is not stated [arxiv:2306.11644 §2.1]. GPT-3.5 writes the textbooks under topic and audience constraints, and writes the exercises as docstrings with reference solutions [arxiv:2306.11644 §2.2]. The block design follows CodeGen, with attention and MLP in parallel, rotary embeddings, 24 layers and hidden size 2048 [arxiv:2306.11644 §2.3]. Pretraining uses AdamW with peak learning rate 1e-3, 750 warm-up steps and weight decay 0.1, on 1024 sequences of 2048 tokens per step, and the step-24,000 checkpoint is used [arxiv:2306.11644 §2.3]. Fine-tuning uses peak learning rate 1e-4 over 6,000 steps [arxiv:2306.11644 §2.3].

Reported effects (author-reported): 50.6% pass@1 on HumanEval and 55.5% on MBPP [arxiv:2306.11644 tab. 1]. The same table lists WizardCoder at 57.3% on HumanEval, above phi-1 (see Contradictions). In a 350M ablation, HumanEval pass@1 rises from 12.19% (unfiltered web code, 96k steps) to 17.68% (filtered, 36k steps) and 20.12% with the synthetic textbooks added [arxiv:2306.11644 §2.1] [fig. 1].

Caveats: the checkpoint criterion for fine-tuning is not named [arxiv:2306.11644 §2.3]. The synthetic data comes from GPT-3.5, which the authors say has a high error rate [arxiv:2306.11644 §6]. The prompts are not published [arxiv:2306.11644 §2.2]. The model is Python-only [arxiv:2306.11644 app. B]. Card: [Textbooks Are All You Need](/papers/2306.11644).

### Autoregressive protein language model (ProGen2)

ProGen2 applies the decoder recipe to protein sequences. A left-to-right Transformer predicts the next residue, in sizes from 151M to 6.4B parameters [arxiv:2206.13517 §3.1] [tab. 1]. The standard models train on UniRef90 mixed with BFD30 [arxiv:2206.13517 §3.2]. A separate antibody model trains on 554M sequences from the Observed Antibody Space, clustered at 85% identity [arxiv:2206.13517 §3.2]. Each sequence gets a terminal tag at each end and is used both as-is and reversed; the weighting between the two orientations is not stated [arxiv:2206.13517 §3.2]. Each block runs attention and the MLP in parallel on one normalised input and uses rotary embeddings [arxiv:2206.13517 §3.1]. Layers, heads, learning rate, batch size, clipping and steps for each size are in Table 1 [arxiv:2206.13517 tab. 1].

Reported effects (author-reported): held-out perplexity on Test-max90 falls from 12.9 to 9.9 as size grows [arxiv:2206.13517 tab. 2]. Zero-shot ranking does not follow size. On the narrow fitness landscapes, base averages Spearman 0.505 and xlarge 0.476 [arxiv:2206.13517 tab. 3] [§4.3]. On antibody general properties, xlarge scores highest at 0.74 [arxiv:2206.13517 tab. 5].

Caveats: the paper defines perplexity but not the zero-shot score behind its fitness tables. The released code averages forward and reversed log-likelihoods, and the paper does not say whether that produced its tables [arxiv:2206.13517 §4.1] [arxiv:2206.13517 §4.3] [code: progen2/likelihood.py#L285-L299]. The release holds inference and scoring code, but no training code or training data [code: progen2/README.md#L8-L14]. The code's default of 28 layers matches no size in Table 1 [arxiv:2206.13517 tab. 1] [code: progen2/models/progen/configuration_progen.py#L33]. Card: [ProGen2: Exploring the Boundaries of Protein Language Models](/papers/2206.13517).

### Molecular SMILES encoder with masked or descriptor pretraining (ChemBERTa-2)

ChemBERTa-2 treats SMILES strings of molecules as text. It is a RoBERTa encoder trained on up to 77M unique PubChem SMILES, in subsets of 5M, 10M and 77M [arxiv:2209.01712 §3]. Two objectives are compared on the same subsets: masked language modeling with 15% of tokens masked [arxiv:2209.01712 §3.1], and multi-task regression on 200 RDKit descriptors computed from each SMILES, each mean-normalised per property [§3.1]. The architecture is searched over 50 random configurations of 5M to 46M parameters, and five configurations per objective are trained on the larger subsets [arxiv:2209.01712 §3.2]. Fine-tuning uses scaffold splits of 80/10/10 on MoleculeNet tasks, for up to 100 epochs with early stopping [arxiv:2209.01712 §3.3].

Reported effects (author-reported, Table 1): the authors find that MTR-pretrained models tend to fine-tune better than MLM-pretrained ones [arxiv:2209.01712 §4.1]. On Lipophilicity RMSE, MTR-10M scores 0.744 against 1.212 for D-MPNN [arxiv:2209.01712 tab. 1]. By the table, ChemBERTa-2 beats D-MPNN on six of eight tasks, and not on BACE classification or ClinTox [arxiv:2209.01712 §4] [tab. 1]. Table 1 gives no units and no spread, so RMSE columns should not be compared across tasks [tab. 1].

Caveats: the 77M corpus, its canonicalisation and the chosen configurations are not in the repository [arxiv:2209.01712 §3.2] [code: chemberta/data/]. The code differs from the paper in places: it divides MTR targets by their standard deviation as well as subtracting the mean [code: chemberta/utils/roberta_regression.py#L218-L219], caps fine-tuning epochs at 10 and fixes the batch size at 64 [code: chemberta/finetune/finetune.py#L92] [#L282-L284]. The repository is MIT-licensed and belongs to a co-author, so it is cited by path. Card: [ChemBERTa-2: Towards Chemical Foundation Models](/papers/2209.01712).

### Motion sequences as words (MotionGPT)

MotionGPT turns human motion into words. A VQ-VAE maps motion to codes from a 512-entry codebook at four-frame downsampling, and the codes join the Flan-T5 vocabulary between start and end tokens [arxiv:2306.14795 §3.1] [§3.2] [§4.1]. The language model is pretrained with span corruption over text and motion, with about 15% of tokens as noise, then instruction-tuned on prompt templates for text-to-motion, motion-to-text, prediction and in-between [arxiv:2306.14795 §3.3].

Reported effects (author-reported, HumanML3D): text-to-motion FID 0.232 after task tuning, against 0.116 for T2M-GPT, which is better on this metric [arxiv:2306.14795 tab. 3]. Motion-to-text BLEU@4 is 12.47, against 7.00 for TM2T [arxiv:2306.14795 tab. 4]. In the paper's ablation, instruction tuning lowers text-to-motion FID at all three sizes [arxiv:2306.14795 tab. 6]. Caveats: the paper says the in-between mask is random, but the released code masks one contiguous block [arxiv:2306.14795 §4.2] [code: mGPT/archs/mgpt_lm.py#L436-L449]; the instruction dataset is promised for release, not released [arxiv:2306.14795 app. G]. Card: [MotionGPT: Human Motion as a Foreign Language](/papers/2306.14795).

### Text-pretrained LM predicting listener gestures (Can Language Models Learn to Listen?)

The MotionGPT pattern applied to listener reactions. Listener facial motion is tokenised by a VQ-VAE into 256 codes. GPT-2 Medium then predicts those codes interleaved with the speaker's timed words, with no audio or video input [arxiv:2308.10897 §3] [app. B]. The language model starts from text-only pretrained weights, while the motion tokens get new embeddings [arxiv:2308.10897 §3.3].

The ablation that isolates initialization differs from the full model only in the pretraining column. On the Trevor Noah split, FD is 22.81 without pretraining and 18.22 with it, and L2 Affect (×10²) is 8.41 against 6.36 [arxiv:2308.10897 tab. 2]. Author-reported, single listener. Interleaving and word order also matter in the same table, for example the unaligned variant reaches FD 19.03 [arxiv:2308.10897 tab. 2]. The model cannot use cues carried by the speaker's face or prosody, such as sarcasm [arxiv:2308.10897 §6.4]. Card: [Can Language Models Learn to Listen?](/papers/2308.10897).

### Multilingual code model with a benchmark that runs tests (CodeGeeX)

CodeGeeX is a 13B left-to-right Transformer trained on code in 23 languages [arxiv:2303.17568 §2.1] [§2.2]. The corpus is 158B tokens, and training runs for 850B tokens over 213,000 steps, which is more than five passes over the corpus [arxiv:2303.17568 §2.2] [§2.3] [tab. 2]. Each segment gets a language tag written in that language's own comment syntax [arxiv:2303.17568 §2.2]. The network is a 39-layer decoder with hidden size 5120 and 40 heads, plus one extra attention layer on top whose query is the embedding of the next position [arxiv:2303.17568 §2.1] [tab. 2] [fig. 2]. The vocabulary has 52,224 entries; runs of spaces get their own tokens on top of a byte-pair tokenizer [arxiv:2303.17568 §2.2]. Training uses Adam with 2000 warm-up steps and cosine decay from a peak of 1e-4 [arxiv:2303.17568 §2.3] [tab. 2].

The authors also built HumanEval-X: 164 problems hand-translated into C++, Java, JavaScript and Go, scored by running unit tests and reported as pass@k with n = 200 samples per problem [arxiv:2303.17568 §3.1] [§3.2] [§4.1]. Reported effects (author-reported): average pass@1 is 18.40% on HumanEval-X generation, against 16.73% for CodeGen-Multi-16B [arxiv:2303.17568 tab. 5]. For inference, INT8 weights with absolute-maximum scaling cut the memory for 128-token generation on A100 from 26.9 GB to 14.7 GB [arxiv:2303.17568 tab. 4] [§2.4]. The authors report that the INT8 model runs on one RTX 3090, where FP16 runs out of memory [arxiv:2303.17568 §2.4].

Caveats: the release's pretraining script differs from the paper on learning rate, batch size, sequence length and Adam β2 [code: scripts/pretrain_codegeex.sh#L94-L113] [arxiv:2303.17568 tab. 2]. The released quantizer takes one maximum per output row, while the paper's equation uses one maximum per layer [code: codegeex/quantization/quantize.py#L55-L56] [arxiv:2303.17568 eq. 4]. The weights are licensed for non-commercial research only [code: MODEL_LICENSE]. Card: [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](/papers/2303.17568).

### Code model on permissively licensed data with fill-in-the-middle (StarCoder)

StarCoderBase is a 15.5B decoder-only code model trained on 1T tokens from The Stack v1.2, which keeps only permissively licensed repositories [arxiv:2305.06161 §3] [§5.5]. It reads 8K tokens with multi-query attention, and character-level fill-in-the-middle (FIM) is applied to half of the code files [arxiv:2305.06161 §5.1] [§5.4]. Near-duplicates are removed with MinHash over 5-grams at Jaccard 0.7, and any file that holds text from an evaluation set is dropped [arxiv:2305.06161 §3.5] [§5.2]. StarCoder is StarCoderBase fine-tuned on about 35B Python tokens [arxiv:2305.06161 §5].

Reported effects (author-reported, pass@1): HumanEval 33.6 and MBPP 52.7 for StarCoder, against 33.5 and 45.9 for code-cushman-001 [arxiv:2305.06161 tab. 12]. StarCoderBase scores 30.4 on HumanEval, below cushman [tab. 12]. Caveats: the public repository holds a LoRA fine-tuning example, not the pretraining or data code [code: finetune/finetune.py#L70-L72]; the SPMv2 layout is named but not defined [arxiv:2305.06161 §5.1]; the weights carry an OpenRAIL-M-type licence, not this paper's CC BY-SA [arxiv:2305.06161 §1]. Card: [StarCoder: may the source be with you!](/papers/2305.06161).

### Surveys

A survey of self-supervised learning places methods in three families by objective: generative (encoder plus decoder, reconstruction or likelihood loss), contrastive (encoder plus a small discriminator, similarity loss such as InfoNCE) and generative-contrastive (a generator with a large discriminator) [arxiv:2006.08218 §2]. It treats masked language modeling as a denoising autoencoder and lists its weakness: masked tokens are assumed independent given the unmasked ones [arxiv:2006.08218 §3.3.3]. Use it as a map of the field. Its numbers are second-hand, and it dates from March 2021 [arxiv:2006.08218 fig. 7].

Card: [Self-supervised Learning: Generative or Contrastive](/papers/2006.08218).

A survey of deep learning for NLP covers pretrained word vectors as the default for rarer words [arxiv:2003.01200 §3.1.3] and lists BERT among models for named entity recognition [arxiv:2003.01200 tab. 6]. It gives no hyperparameters or code, and its score tables collect numbers from other papers with differing metrics and setups, so do not rank methods from them [arxiv:2003.01200 tab. 2–7].

Card: [Natural Language Processing Advancements By Deep Learning: A Survey](/papers/2003.01200).

A survey of large models for time series and spatio-temporal data takes a different angle. It sorts work into a time-series branch and a spatio-temporal branch, and each branch into language-model and foundation-model approaches, then into general and domain-specific work [arxiv:2310.10196 §3] [arxiv:2310.10196 fig. 3]. It has no experiments, and its method table lists category, scope, task and venue for each surveyed method [arxiv:2310.10196 tab. 2]. The authors call the language-model and foundation-model split practical rather than strict, and they note that foundation models for time series are still at an early stage [arxiv:2310.10196 §3]. Card: [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](/papers/2310.10196).

A survey of prompt-based methods rewrites supervised NLP as filling a blank in a pretrained model's input. It has three steps: add the prompt, search for the answer words, and map the answer words to labels [arxiv:2107.13586 §2.2]. Its taxonomy covers prompt engineering, answer engineering, multi-prompt learning and training strategies [arxiv:2107.13586 §2.3]. It reports no numbers of its own [arxiv:2107.13586 §11].

Card: [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](/papers/2107.13586).

A survey of pretrained language models for NLP sorts their use into three paradigms: pre-train then fine-tune, prompt-based learning, and NLP as text generation [arxiv:2111.01243 §2] [§3] [§4]. For staged fine-tuning it describes a common practice rather than a fixed recipe: keep the pretrained layers frozen while the new layers train, then unfreeze them gradually with a learning rate between 1e-3 and 1e-5 [arxiv:2111.01243 §2.4.2]. It runs no experiments of its own [arxiv:2111.01243 §6]. Card: [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](/papers/2111.01243).

A survey of large language models covers how they are built (tokenisers, positional encodings, pretraining objectives, alignment and decoding), how they are used with prompts, retrieval and agents, and the datasets and metrics used to score them [arxiv:2402.06196 §3] [§4] [§5]. It proposes no method and runs no experiments. Its benchmark tables are compiled from other papers, so compare numbers only within one row [arxiv:2402.06196 §6.2]. Card: [Large Language Models: A Survey](/papers/2402.06196).

A review of BERT in NLP covers what came after the original model: retraining on domain text, extractive sentence selection, BERTScore, black-box attacks on BERT classifiers, and cross-lingual and domain variants [arxiv:2103.11943 p. 1]. It reports that further training on in-domain text helps, and helps more when the text is on the same subject, but that it can cause forgetting unless the learning rate is low [arxiv:2103.11943 p. 3]. It reports no new experiments. Its tables quote numbers from the papers it cites, such as the SciBERT comparison [arxiv:2103.11943 p. 11], and its method descriptions are paraphrases, so check numbers and formulas against the originals [arxiv:2103.11943 p. 6]. Card: [BERT: A Review of Applications in Natural Language Processing and Understanding](/papers/2103.11943).

A survey of large language models with 10B or more parameters covers building blocks, pretrained and tuned models, efficiency methods, datasets, evaluation and applications [arxiv:2307.06435 §1] [§2] [§3]. It tabulates architecture and optimizer settings for models such as LLaMA, PaLM and BLOOM, and many of its cells are blank or use undefined symbols, so check each row against its source [arxiv:2307.06435 tab. 5] [tab. 6]. It runs no experiments of its own, and its top-result table is compiled from cited papers [arxiv:2307.06435 §5]. Card: [A Comprehensive Overview of Large Language Models](/papers/2307.06435).

## Effects of the pretraining corpus

### Rare facts

Closed-book question answering accuracy rises with the number of pretraining documents that mention both a question's entity and its answer. The counts come from entity linking over The Pile, ROOTS, C4, OpenWebText and Wikipedia [arxiv:2211.08411 §2, §3.1]. For BLOOM-176B, questions with about 10 relevant documents score 25%, and those with about 10,000 score above 55% [arxiv:2211.08411 §3.1, fig. 1]. Removing the matching C4 documents and retraining a 4.8B model lowers accuracy on those questions. The authors read this as causal, though the text gives no numbers [arxiv:2211.08411 §3.2, fig. 5]. Gains from scale on rare questions are log-linear. A log-linear fit with R² of 0.98 projects more than 10^18 parameters before a supervised baseline or humans would be matched [arxiv:2211.08411 §4.2, fig. 6]. Retrieval helps far more: a BM25 setting that adds the top three Wikipedia paragraphs beats closed-book GPT-Neo at every count, most of all on rare questions [arxiv:2211.08411 §4.3, fig. 9]. The count is only a proxy. In a human check of 300 sampled pairs, about a third of the relevant documents held enough to answer the question [arxiv:2211.08411 §2.3]. GPT-3's counts are estimates scaled from OpenWebText, not measurements [arxiv:2211.08411 §3]. Card: [Large Language Models Struggle to Learn Long-Tail Knowledge](/papers/2211.08411).

### Political lean

Pretrained models carry a political lean, and continued pretraining on partisan text moves it. The authors probe 14 models with a 62-statement political compass test [arxiv:2305.08283 §2.1, §3]. Continued pretraining of RoBERTa-base on reddit-left text moves its social score from 2.97 to -3.03, while most other shifts are small [arxiv:2305.08283 §4.1, fig. 3]. Downstream, RoBERTa variants trained on left or right corpora differ on hate-speech and misinformation detection. Overall F1 gaps reach about 3 points, while the differences between identity groups and news sources are the main finding [arxiv:2305.08283 tab. 3, tab. 4, §4.2]. Vanilla RoBERTa is the top row on several overall metrics [arxiv:2305.08283 tab. 3]. The partisan checkpoints are withheld, and the authors describe training them as a dual-use risk, according to their ethics statement. Card: [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](/papers/2305.08283).

### Documented multi-source English corpus (The Pile)

The Pile is a fixed mixture of 22 English sources, about 825 GiB in total. Some sources are given larger weights, so their text is seen more than once in training [arxiv:2101.00027 §1]. The authors report that 1.3B models trained on the Pile beat models trained on CC-100 and raw Common Crawl, on the Pile's own components and on a standard word-prediction benchmark [arxiv:2101.00027 §1]. The paper documents bias, profanity and consent issues for each source, and several sources carry terms-of-service caveats, so check the documentation before reuse [arxiv:2101.00027 §6].

Card: [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](/papers/2101.00027).

### Document-level web corpus construction (OSCAR 22.01)

OSCAR 22.01 rebuilds a multilingual web corpus from one Common Crawl dump so that each document gets one language label, rather than one label per line [arxiv:2201.06642 §4]. The steps are: trim short lines at the head and tail, drop a document when short lines dominate, identify the language of each line, test whether the document is multilingual, and otherwise keep it under its dominant language if the byte-weighted confidence reaches 0.6 [arxiv:2201.06642 §3] [arxiv:2201.06642 §4] [arxiv:2201.06642 §4.2]. Quality is recorded as annotations rather than by deletion: tiny, short_sentences, header, footer, noisy and adult [arxiv:2201.06642 §5].

Reported effects (author-reported): mid- and high-resource corpus sizes stay close to OSCAR 21.09, while several low-resource corpora shrink [arxiv:2201.06642 §6.1]. The authors do not establish the intrinsic quality of unannotated documents [arxiv:2201.06642 §6.2.3]. In a hand check of the first 100 French documents flagged as adult, 77 were true positives and 23 were false positives, 2 of them on LGBTQI+ sites [arxiv:2201.06642 §6.2.4].

Caveats: the released code differs from the paper. Its multilingual test needs 10 lines, not 5, and its head and tail trim removes every line before the first long line and after the last one. Its filters also count length in characters in one place and bytes in another [code: src/identifiers/multilingual.rs#L191] [code: src/transformers/sentence_filter.rs#L190-L202] [code: src/transformers/header.rs#L76-L86]. Card: [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](/papers/2201.06642).

### Documenting a web-scraped corpus (C4)

C4 is the cleaned April 2019 Common Crawl snapshot used to train T5. Its English version, C4.en, has 365 million documents, 156 billion tokens and 305 GB [arxiv:2104.08758 §2] [tab. 1]. The paper documents three views of it: where the text comes from, what it says, and what the filters removed [arxiv:2104.08758 §1]. Patents.google.com is the most represented website [arxiv:2104.08758 §3.1]. In a sample of 1,000,000 URLs, 92% were first indexed by the Internet Archive between 2011 and 2019, which is an estimate, since the archive date is only a proxy for when a page was written [arxiv:2104.08758 §3.2].

Benchmark test items occur verbatim in the corpus, after case and punctuation normalisation: from 1.87% to 24.88% of generation targets, and from 1.8% to 53.6% of GLUE-style inputs [arxiv:2104.08758 §4.2] [tab. 4]. The bad-word blocklist removes 42% of documents classed as African American English and 32% of Hispanic-aligned ones, against 6.2% and 7.2% for White-aligned and other English [arxiv:2104.08758 §5.3]. The authors advise against blocklist filtering when building a web corpus [arxiv:2104.08758 §6]. A sentiment check finds that "Jewish" co-occurs with more positive words than "Arab" in C4: a 7.5-point gap with one lexicon, and 1.4 and 2.0 points with two others [arxiv:2104.08758 §4.3] [app. A.7].

The paper is a documentation study with no model and no training method. The linked repository holds only a README, with no filter or analysis code and no licence file [code: README.md]. The corpus likely contains personal information and copyrighted text, which the authors did not quantify [arxiv:2104.08758 §6]. Use the paper as a checklist for documenting a corpus, not as a licence for the data. Card: [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](/papers/2104.08758).

## Evaluation and cost

### Robustness under perturbation

Adversarial GLUE is a test set of adversarial examples for five GLUE tasks: SST-2, QQP, MNLI, QNLI and RTE [arxiv:2111.02840 §3.1]. The examples come from 14 attack entries, run against BERT and RoBERTa surrogates, then filtered for transfer and fidelity. Crowd workers check what remains, and an example is kept only when four of five annotators agree with the original label [arxiv:2111.02840 §3.2, §3.3]. Across ten models, including robust-training variants, the average AdvGLUE score ranges from 30.3 to 60.9. On GLUE the same models score 85.7 to 93.2 [arxiv:2111.02840 tab. 4]. DeBERTa (Large) is highest on AdvGLUE, and BERT (Large) sits near the bottom. ELECTRA (Large) drops from 93.16 on GLUE to 41.69 [arxiv:2111.02840 tab. 4]. Only the dev split is public, and test scores come from a hidden set on CodaLab [arxiv:2111.02840 §3.5]. Card: [Adversarial GLUE](/papers/2111.02840).

### Pretraining under a word budget (BabyLM)

The BabyLM Challenge caps pretraining text at about 100M English words in its Strict and Loose tracks, and at 10M in Strict-Small [arxiv:2504.08165 §3] [arxiv:2504.08165 tab. 1]. Models are scored zero-shot on minimal-pair grammar tests (BLiMP and a hidden supplement), finetuned on a GLUE subset, and finetuned on MSGS [arxiv:2504.08165 §5.1]. The leaderboard aggregate weights grammar at 50%, GLUE at 30% and MSGS at 20% [arxiv:2504.08165 §5.3]. Repeating the same text over several epochs does not count against the budget, so the strongest Strict entry trained for hundreds of epochs [arxiv:2504.08165 §3] [arxiv:2504.08165 §7.2].

Reported effects (author-reported): the LTG-BERT-based ELC-BERT leads the Strict track with aggregate 0.74, against 0.60 for the best baseline, OPT-125M [arxiv:2504.08165 tab. 2]. In Strict-Small it reaches 0.66, against 0.50 for OPT-125M [arxiv:2504.08165 tab. 2]. Most curriculum-learning entries did not help. Architecture changes, training objectives, data preprocessing and teacher–student distillation gave the larger gains [arxiv:2504.08165 §7.4].

Caveats: the MSGS column in Table 2 is positive, while the appendix reports mostly negative MCC values for the same models, and the paper does not explain the conversion [arxiv:2504.08165 tab. 2] [arxiv:2504.08165 app. D]. Some submissions go past the word budget through augmentation. The Loose winner builds about 4B words from 100M [arxiv:2504.08165 §7.2]. Card: [Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](/papers/2504.08165).

### Energy and carbon of a whole project

Pretraining energy is only part of a model project's cost. For Noor, a family of 1.3B to 13B Arabic models trained on 150B tokens each, the authors add data storage and processing, R&D, the four training runs, staff travel and projected inference [arxiv:2610.00223 §3, §5, §5.1]. The project used 59.14 MWh of electricity, and the authors estimate 36.5 t CO2e. Training accounts for 65% of the emissions and flights for 18% [arxiv:2610.00223 fig. 2, fig. 3]. Location matters a great deal: the authors estimate 14.9 t if all the compute had run in France [arxiv:2610.00223 §7]. The inference figures are a prospective estimate that assumes batch size 1 on one A100 [arxiv:2610.00223 §5.1.5]. The paper's equivalence between token count and training energy does not follow from its own per-token figure, by our arithmetic in the card [arxiv:2610.00223 §5.1.5], so use the per-token value instead. Card: [A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model](/papers/2610.00223).

### Pretraining quality and the fine-tuning dimension

Fine-tuning can be reparameterised so that only a small number d of trainable values is learned, with a fixed random projection into the full weight space [arxiv:2012.13255 §3] [arxiv:2012.13255 eq. 1] [arxiv:2012.13255 eq. 2]. For RoBERTa-Large on MRPC, about 200 such values reach 90% of full fine-tuning [arxiv:2012.13255 §4.2] [arxiv:2012.13255 tab. 1]. The smallest d is lower for models pretrained longer [arxiv:2012.13255 §5.1] [arxiv:2012.13255 fig. 2] and for larger models [arxiv:2012.13255 §5.2] [arxiv:2012.13255 fig. 3]. The authors also link a lower d to better generalisation [arxiv:2012.13255 §5.3] [arxiv:2012.13255 fig. 4]. All numbers are author-reported.

Card: [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](/papers/2012.13255).

## When to use which

The cards target different regimes and never compare on a shared setup, so the table below maps constraints to the recipe each card supports rather than ranking them.

| Situation | Use | Why |
|---|---|---|
| Labeled data per task, encoder-style NLU (classification, extractive QA, ranking) | Bidirectional pretraining, e.g. permutation LM, then fine-tune | Beats BERT-Large under matched data and hyperparameters on 11/11 tasks [arxiv:1906.08237 tab. 1]; fine-tuning recipes given for RACE, SQuAD, MNLI, Yelp-5 [arxiv:1906.08237 tab. 8] |
| One language, enough text in that language, want an encoder | Monolingual masked LM with whole-word masking (CamemBERT's recipe) | Beats the multilingual XLM-R BASE on French XNLI with fewer parameters [arxiv:1911.03894 tab. 4]; tested on French only [arxiv:1911.03894 §1] |
| French, want a base or large encoder and a multi-task French benchmark | FlauBERT's recipe (MLM on 71 GB cleaned French text) | Large beats mBERT on most FLUE tasks [arxiv:1912.05372 §5]; CamemBERT is ahead on PAWS-X [arxiv:1912.05372 tab. 4]; the authors' code is non-commercial, so check the weights and corpora before commercial use [arxiv:1912.05372 §3.1] |
| Limited raw text in the target language or domain | Diverse web text rather than Wikipedia alone | At equal size, Common Crawl-derived text beat Wikipedia; 4GB of web text came close to 138GB at a fixed step budget [arxiv:1911.03894 §6.1, §6.2, tab. 5] |
| Labeled span data (aspect terms, entities) and an encoder you can fine-tune | Fine-tune BERT-base with one linear head; add a richer head only if the gain is worth it | Linear head beats the best non-BERT baseline by a small margin (60.43 vs 60.35 F1 on LAPTOP); richer heads add at most 1.5 F1 on REST [arxiv:1910.00883 tab. 2] |
| Structured event histories (diagnoses, visits) with outcomes at fixed horizons, and licensed access to the data | BEHRT's recipe: masked prediction over codes with age and visit embeddings, then one head per horizon | Beats DeepR and RETAIN on APS at three horizons (author-reported) [arxiv:1907.09538 tab. 1]; you must build your own code vocabulary [arxiv:1907.09538 §2.2] |
| Fixed embeddings for scientific papers, no per-task training, citation graph available for training | SPECTER's recipe: SciBERT plus a citation triplet loss | Average 80.0 vs 59.6 for SciBERT on SciDocs [arxiv:2004.07180 tab. 1]; inference needs only title and abstract [arxiv:2004.07180 §2.5] |
| You need an open, self-hostable multilingual generator and can run a multi-month job on hundreds of GPUs | BLOOM's recipe: causal LM with ALiBi and an embedding LayerNorm | Released openly, trained on 384 A100s for about 3.5 months [arxiv:2211.05100 §3.4]; the ALiBi rule differs from the code for 112 heads, so check the reference implementation before copying it [arxiv:2211.05100 §3.2] [code: megatron/model/transformer.py#L596-L607] |
| Many tasks, no per-task training, generation needed | Causal decoder, few-shot prompting | PaLM's whole evaluation is 0/1/few-shot [arxiv:2204.02311 §6.1]; the AR factorization generates text directly |
| You can fine-tune and care about peak accuracy on a fixed task | Fine-tune, even a large decoder | PaLM's own few-shot scores trail its fine-tuned scores on SuperGLUE dev, e.g. WiC 64.6 vs 78.8, RTE 81.2 vs 95.7 [arxiv:2204.02311 tab. 8] |
| Multi-document question answering, with links or citations between documents in the corpus | Pretrain on linked document pairs with a three-way relation loss (LinkBERT recipe), then fine-tune | MRQA average F1 over BERT: 77.8 against 75.2 at base and 81.0 against 78.5 at large [arxiv:2203.15827 tab. 1]; random links give no gain at tiny size [arxiv:2203.15827 §5.5] |
| Long documents in inputs | Segment recurrence / memory (Transformer-XL backbone) | Removing memory costs most on RACE [arxiv:1906.08237 tab. 6] |
| Summarization or question answering over inputs of several thousand tokens, with a text-to-text model | Sparse encoder attention with transient global tokens and principle-sentence pretraining (LongT5 recipe) | PubMed ROUGE-1 50.23 for xl at 16k against 48.36 for HAT-BART, author-reported [arxiv:2112.07916 tab. 2]; the input length differs per dataset, so the comparison is not controlled [arxiv:2112.07916 tab. 2] |
| Want bidirectional context but no `[MASK]` pretrain/fine-tune mismatch | Permutation LM with two-stream attention | Objective uses no input corruption and keeps the exact product-rule factorization [arxiv:1906.08237 §2.2, §2.6] |
| Adapting a pretrained model to an aspect-level task | Reformulate the task for the model (sentence pair, reading comprehension, generation) | The survey reports this as the main line of recent work [arxiv:2203.01054 §5]; one cited evaluation found a drop of more than 25% on adversarial examples for the simplest ASC task [arxiv:2203.01054 §5] |
| Low implementation effort | Plain causal LM | Permutation LM needs two attention masks, a query stream and target sampling, and loose spots in the paper must be filled from code [arxiv:1906.08237 §2.3, §3.1]; a causal LM needs one causal mask [arxiv:2204.02311 §2] |
| Inference memory / decoding cost matters | Multi-query attention | Reported neutral on quality and training speed with large decoding savings [arxiv:2204.02311 §2] |
| Training throughput at very large scale | Parallel blocks, pipeline-free sharding, track MFU | ~15% faster training from parallel blocks [arxiv:2204.02311 §2]; 46.2% MFU without pipeline parallelism [arxiv:2204.02311 tab. 3] |
| Reasoning benchmarks (math word problems) | Largest feasible decoder plus chain-of-thought prompting | GSM8K 17% without CoT vs 54% with CoT for 540B, 33% for 62B with CoT [arxiv:2204.02311 tab. 10] |
| Fixed compute budget, choosing size vs tokens | More tokens for mid-size models is worth testing | 62B trained to 1325B tokens improved strongly on MMLU and GSM8K [arxiv:2204.02311 app. F]; refreshed data beat repeated data on most of 8 tasks [arxiv:2204.02311 app. F] |
| Adapting a model to a private domain with a large in-house corpus | Continued pretraining from a base model with an extended tokenizer, then short fine-tuning (ChipNeMo recipe) | Full continued pretraining beat LoRA on in-domain tasks [arxiv:2311.00176 §3.4]; the 70B run cost 20,500 A100-hours [arxiv:2311.00176 §3.5] |
| Short social-media text in one domain, a BERT-Large-scale encoder, and an unlabelled in-domain corpus of about 22.5M posts | Continued masked-LM and next-sentence training of BERT-Large on in-domain text, then fine-tuning (CT-BERT recipe) | Mean F1 0.833 against 0.802 for BERT-Large on five sets, author-reported [arxiv:2005.07503 tab. 2]; the run took 120 hours on a TPU v3-8 [arxiv:2005.07503 §2] |
| Large medical or other expert domain with a 7B or 70B open base, and a limited share of general text to keep | Continued next-token pretraining with a general-text replay share, then one fine-tune per benchmark (MEDITRON recipe) | MEDITRON-70B with self-consistency averages 72.0 against 69.2 for fine-tuned Llama-2-70B [arxiv:2311.16079 tab. 5]; the 70B run took 332 hours on 128 A100s [arxiv:2311.16079 app. A] |
| Transductive text classification over a fixed corpus that includes unlabelled documents | Encoder inside a word-document graph (BertGCN recipe) | Above BERT alone on all five benchmarks, author-reported [arxiv:2105.05727 tab. 1]; needs a memory bank and a small BERT learning rate [arxiv:2105.05727 §3.3] |
| Search or retrieval over paired text or code, where similarity benchmarks matter less | Contrastive embedder with in-batch negatives and very large batches | Beats BM25 on MSMARCO and Contriever on NQ and TriviaQA [arxiv:2201.10005 tab. 5]; trails SimCSE on sentence similarity [arxiv:2201.10005 tab. 4] |
| Open-ended generation where named entities must be correct | Title-prefixed continued training with a sentence-completion loss, plus factual-nucleus decoding | At 530B, named-entity error on factual prompts falls from 33.3% to 14.5%, with lower diversity [arxiv:2206.04624 tab. 5]; Wikipedia-only training did not help at 1.3B [arxiv:2206.04624 tab. 5] |
| Web agent trained from recorded human sessions, with a pretrained language model available | Imitation of search and choice models, then policy gradient on the choice model | Imitation plus RL scores 62.4 against 45.6 for the rule baseline [arxiv:2207.01206 tab. 2]; RL from raw BERT starts below the rule [arxiv:2207.01206 §5.2] |
| Facts that are rare in the web text the model saw | Retrieve passages at answer time rather than expect more pretraining to store them | Scale helps rare questions only log-linearly [arxiv:2211.08411 §4.2]; BM25 retrieval beats closed-book at every count [arxiv:2211.08411 §4.3] |
| Checking a pretraining corpus for political lean before downstream use | Probe the model with statements, then test downstream results per group | Overall gaps are about 3 F1 points; group and source differences are the main finding [arxiv:2305.08283 §4.2] |
| Reporting robustness of a fine-tuned encoder | Evaluate on adversarial test sets as well as GLUE | Average AdvGLUE score 30.3 to 60.9, against 85.7 to 93.2 on GLUE [arxiv:2111.02840 tab. 4] |
| Budgeting energy and carbon for a model project | Count data, R&D, travel and inference, not only training | Training was 65% of emissions and flights 18% for Noor [arxiv:2610.00223 fig. 3]; all compute in France would give 14.9 t [arxiv:2610.00223 §7] |
| Non-text sequences (bits, pixels, protein sequences), little compute to adapt, and a language-pretrained transformer to reuse | Frozen-block recipe: freeze attention and feedforward, train input and output layers, positions and layer norms | Layer norms alone give 54% on CIFAR-10 against 25% for the output layer alone [arxiv:2103.05247 tab. 15]; frozen random LSTMs do much worse [arxiv:2103.05247 tab. 3]; the code's trained subset differs from the paper in places [arxiv:2103.05247 §2.2] |
| Task-oriented dialogue with sub-task labels that exist for only some corpora | Text-to-text pretraining with a prompt per sub-task, then fine-tuning with the same objective | 1% DST joint accuracy 31.46 against 13.21 for SOLOIST [arxiv:2109.14739 tab. 5]; at 10 samples per intent, USE+ConveRT is ahead [arxiv:2109.14739 tab. 7] |
| Few-shot use in many languages, including low-resource ones, with no fine-tuning and no labelled data | Causal LM over a multilingual corpus, scored by label likelihood | mGPT-1.3B is close to XGLM-1.7B on classification [arxiv:2204.07580 tab. 7]; 4-shot NER and POS are far below fine-tuned encoders [arxiv:2204.07580 tab. 9] [arxiv:2204.07580 tab. 10] |
| Decoder for general use, public data mix, about 1T to 1.4T tokens | LLaMA-style recipe: pre-norm RMSNorm, SwiGLU, rotary embeddings, public corpus, cosine decay to 10% | Trained on public data only [arxiv:2302.13971 §2.1]; LLaMA-13B beats GPT-3 on most common-sense tasks, author-reported [arxiv:2302.13971 §3.1] |
| Chat assistant from your own base model, with a reward model trained on preference pairs and PPO | Qwen-style alignment: ChatML SFT, reward model from a same-size base, PPO with a pretraining-gradient term | Reward model reaches 74.78 against 62.68 for the PMP baseline on Helpful-base, author-reported [arxiv:2309.16609 tab. 4]; PPO batch size and step count are not stated [arxiv:2309.16609 §3.2.2] |
| Inputs longer than the training context, with no training budget for extension | Inference-time context extension (NTK, LogN, window attention) | The paper gives no formulas and no per-layer window sizes, so exact implementation needs the released code [arxiv:2309.16609 §2.5]; the author-reported arXiv perplexity at 8192 tokens stays low with all techniques [arxiv:2309.16609 tab. 3] |
| Study how behaviour changes over training, with a reproducible run | Pythia-style suite: same data order, all checkpoints, released code | Eight sizes with 154 checkpoints each [arxiv:2304.01373 §1] [arxiv:2304.01373 §2.4] |
| Public decoder where the run must be auditable, including failures | OPT-style logbook: GPT-3-like settings plus a record of restarts and loss spikes | At least 35 manual restarts recorded [arxiv:2205.01068 §2.5]; a lower launcher learning rate than the paper's table [arxiv:2205.01068 tab. 1] |
| Large total capacity with low per-token compute, and serving cost matters | Sparse MoE with top-2 routing in every other block (GLaM) | 1.2T total parameters, 96.6B active per token [arxiv:2112.06905 tab. 4]; about half of GPT-3's inference FLOPs [arxiv:2112.06905 tab. 1] |
| Knowledge-heavy generation where a large text store is available and parameters should hold less | Chunked retrieval with frozen BERT keys (RETRO), or RETRO-fitting a frozen model | Retrieval-fitting trains only the new layers [arxiv:2112.04426 §4.2]; 7.5B beats Jurassic-1 and Gopher on most Pile test sets [arxiv:2112.04426 §4.1] |
| German-only encoder, with a filtered web corpus to test | GottBERT's RoBERTa recipe on German OSCAR, base or large | Base leads on 4 of 6 tasks, with margins up to 0.85 points and no significance test [arxiv:2012.02110 tab. 3] |
| Code completion inside an editor, with infilling | Llama 2 continued on code with an infilling objective and a long-context stage (Code Llama recipe) | Infilling costs 0.6 points at 7B on average [arxiv:2308.12950 §3.2]; HumanEval up to 67.8 for 70B Instruct, author-reported [arxiv:2308.12950 tab. 2] |
| Small model on a phone or a tight memory budget | Data-quality-first recipe: filtered web data, synthetic reasoning data, SFT then DPO (Phi-3) | 3.8B model at more than 12 tokens per second on an iPhone 14 at 4-bit [arxiv:2404.14219 §2] [arxiv:2404.14219 fig. 2] |
| Small open family for benchmark-heavy use | Gemma-style: larger-model recipe scaled down, instruction and RLHF tuning | 7B averages 56.9 over 18 benchmarks, against 54.5 for Mistral 7B [arxiv:2403.08295 tab. 6] |
| Multilingual model where compute is fixed and the data-to-size ratio is open | Compute-optimal scaling with equal growth of data and parameters, plus a multilingual mix (PaLM 2) | BIG-Bench Hard 78.1 against 65.2 for PaLM [arxiv:2305.10403 tab. 6]; sizes withheld [arxiv:2305.10403 app. D] |
| Adapting a pretrained model with very few trainable values | Fine-tune in a random low-dimensional subspace | About 200 values reach 90% of full fine-tuning on MRPC [arxiv:2012.13255 tab. 1]; better-pretrained models need fewer [arxiv:2012.13255 §5.1] |
| Entity facts matter for the task and a knowledge graph covers the domain; an encoder is enough | Joint text and subgraph encoder with MLM plus link prediction | 76.0 on CSQA with both objectives, against 74.3 with MLM alone [arxiv:2210.09338 tab. 5]; 47.5 on MedQA [arxiv:2210.09338 tab. 6]; the model does not generate text [arxiv:2210.09338 §5] |
| Building a multilingual pretraining corpus from one Common Crawl dump | Document-level language ID, annotations instead of deletion, then filter on annotations | Mid- and high-resource sizes stay near OSCAR 21.09 [arxiv:2201.06642 §6.1]; quality is not established [arxiv:2201.06642 §6.2.3] |
| Pretraining budget of about 10M to 100M words | Encoder with the LTG-BERT design; do not expect curriculum learning to help by default | ELC-BERT reaches 0.74 aggregate on Strict against 0.60 for OPT-125M [arxiv:2504.08165 tab. 2]; most curriculum entries did not help [arxiv:2504.08165 §7.4] |
| Spanish-only encoder, with enough Spanish text and no need for other languages | BETO recipe: BERT-Base with whole-word and dynamic masking on about 3B Spanish words | Beats the best multilingual BERT on XNLI (82.01 against 78.50, cased) and NER (88.43 against 87.38) [arxiv:2308.02976 tab. 1]; trails it on TAR-to-XQuAD F1 [arxiv:2308.02976 tab. 2]; training-data licences need checking [code: README.md#L58-L59] |
| Questions over tables, where a parser or encoder must read cell values and column types | Table-text pretraining over K sampled rows with vertical attention (TaBERT) | 64.5 against 61.3 for BERT-Large on Spider dev, author-reported [arxiv:2005.08314 tab. 2]; the released code is non-commercial, so write the encoder from the paper [code: LICENSE.md] |
| Python code model with a few billion parameters or fewer, and a corpus you can filter and generate | Filtered code plus synthetic textbooks and exercises (phi-1 recipe) | 50.6% HumanEval and 55.5% MBPP at 1.3B, author-reported [arxiv:2306.11644 tab. 1]; WizardCoder is higher on HumanEval in the same table [arxiv:2306.11644 tab. 1] |
| Building a web corpus, or auditing one you did not build | Publish sources, contamination and removed-text analyses; do not rely on word blocklists | Blocklist removal of 42% for African American English against 6.2% for White-aligned English [arxiv:2104.08758 §5.3]; the authors advise against blocklists [arxiv:2104.08758 §6] |
| Orientation to pretraining, fine-tuning or LLM use before choosing a method | Read a survey first, then the primary paper before implementing | Both surveys run no experiments [arxiv:2111.01243 §6] [arxiv:2402.06196 §6.2]; their tables are not rankings [arxiv:2402.06196 §6.2] |
| Protein sequences, where scoring or generating variants needs no alignment and no labels | Decoder-only next-token model, with a size sweep, since ranking did not improve with scale | Narrow-landscape average Spearman 0.505 for base against 0.476 for xlarge [arxiv:2206.13517 tab. 3]; ranking peaks at 764M parameters [arxiv:2206.13517 §4.3] |
| Code generation or translation across several languages, with open weights and limited GPU memory | Decoder trained on language-tagged code, with INT8 weights for inference (CodeGeeX recipe) | Average pass@1 18.40% against 16.73% for CodeGen-Multi-16B on HumanEval-X [arxiv:2303.17568 tab. 5]; the weights are non-commercial, so check the licence first [code: MODEL_LICENSE] |
| Arabic text with dialects, compared against multilingual encoders | ARBERT and MARBERT recipe: a formal-Arabic encoder and a tweet-trained one, evaluated on ARLUE | MARBERT-v2 scores 77.40 against 76.55 for XLM-R Large, author-reported [arxiv:2101.01785 §5.2]; the tweet pretraining data is in-house and not released [arxiv:2101.01785 §3.2] |
| Small labelled Arabic tweet set with rare classes (e.g. sarcasm), where you can collect more examples | Fine-tune MARBERT, and collect more of the rare class rather than resample the existing data | Sarcasm F1 0.33 at training batch 16 and 0.54 at batch 96, author-reported [arxiv:2606.25495 tab. 5]; the paper runs no baseline on its private data [arxiv:2606.25495 tab. 2] |
| English understanding fine-tuning, where every token should train the model | Replaced token detection with gradient-disentangled embedding sharing (DeBERTaV3 recipe) | GLUE dev average 91.37 for large against 90.00 for DeBERTa-large, author-reported [arxiv:2111.09543 tab. 3]; the discriminator trains on every token [arxiv:2111.09543 §2.3.1] |
| Large bilingual model that must run on a few consumer GPUs | GLM-130B's weight-only INT4 quantization with per-row absolute-maximum scales | LAMBADA 79.47 at INT4 against 80.21 at FP16 [arxiv:2210.02414 §4]; the released weights are non-commercial [code: MODEL_LICENSE] |
| Scientific text where equations, citations or step-by-step working must come out right | Galactica's curated corpus with citation, working-memory and molecule tokens | LaTeX equation recall 68.2% against 49.0% for GPT-3, author-reported [arxiv:2211.09085 tab. 6]; the repository gives CC BY-NC 4.0 for the 120B weights [code: LICENSE-MODEL.md] |
| Chat model that must choose among tools by itself, in Chinese and English | GLM-4 alignment with model-chosen tools (All Tools recipe) | GSM8K Python task 91.59 against 92.72 for GPT-4 (Web), author-reported [arxiv:2406.12793 tab. 9]; the tool-choice training method is not stated [arxiv:2406.12793 §2] |
| Code completion or infilling in an editor, with permissively licensed training data | StarCoder recipe: permissive code, character-level FIM at rate 0.5, multi-query attention, 8K context | HumanEval 33.6 and MBPP 52.7 against 33.5 and 45.9 for code-cushman-001 [arxiv:2305.06161 tab. 12]; the weights carry an OpenRAIL-M-type licence [arxiv:2305.06161 §1] |
| One model for several motion tasks, with motion stored as discrete codes | VQ-VAE motion codes as vocabulary words, span corruption, then instruction tuning (MotionGPT recipe) | Text-to-motion FID 0.232 on HumanML3D, behind T2M-GPT at 0.116 [arxiv:2306.14795 tab. 3]; the claim is breadth across tasks, not the best FID [arxiv:2306.14795 tab. 3] |
| Predict a listener's facial reactions from the speaker's transcript alone, with no audio or video | Text-pretrained GPT-2 Medium that emits 256 VQ motion codes interleaved with timed words | FD 18.22 with text pretraining against 22.81 without, Trevor Noah split, author-reported [arxiv:2308.10897 tab. 2]; human A/B preference over retrieval 70.1% in a study on 47 curated moments [arxiv:2308.10897 §5.2] |
| Short social-media text in several languages where a BERT exists for each, with a binary label and a dev split | Language-specific BERT-base with a CNN over its last four layers (KUISAIL recipe), or plain fine-tuned BERT if the CNN adds nothing for your languages | Average macro-F1 0.851 against 0.841 for BERT alone, but Turkish is 0.002 lower, author-reported from single runs [arxiv:2007.13184 tab. 2]; the paper does not give a loss or threshold [arxiv:2007.13184 §4.4] |
| Long causal inputs of several thousand tokens, where training cost must grow close to linearly and you can accept a non-standard attention block | Gated single-head attention with exact attention in chunks and summaries across them (FLASH recipe) | Training-cost speedup up to 4.9x on Wiki-40B at context 8192 against Transformer++, author-reported [arxiv:2202.10447 §4.2]; the equations and appendix listings differ on scaling and masks, so test both [arxiv:2202.10447 app. D] |
| Dutch text only, BERT-base size, and you want an encoder with a documented corpus recipe | BERTje recipe: sentence-order prediction and about 12 GB of Dutch text, then fine-tune per task | NER test F1 88.3 against 80.7 for multilingual BERT, author-reported [arxiv:1912.09582 tab. 1]; the released code masks single wordpieces, not the whole words the paper describes [code: pretraining/create-pretrain-data.py#L32] |
| Domain text whose vocabulary general models miss, a few hundred million words of in-domain text, and a labelled entity set | Continue masked LM from a science-domain model on in-domain full text, then fine-tune with a CRF or BiLSTM-CRF head (MatSciBERT recipe) | SOFC-Slot Macro-F1 65.92 against 61.68 for SciBERT, author-reported [arxiv:2109.15290 tab. 2]; the pretraining corpus is not released [arxiv:2109.15290 §2.1] |
| Multilingual classification where misspellings, code-switching and obfuscation are common, and per-language vocabularies are a burden | Token-free byte-level encoder with a learned byte-block tokenizer and span-denoising pretraining (UTC recipe) | Production-Multilingual average AUC-ROC 0.910 (102M) and 0.919 (268M), author-reported [arxiv:2202.11176 tab. 1]; the 102M model trails mT5-base on code-switched JMTCC (0.9191 against 0.9393) [arxiv:2202.11176 tab. 3] |
| Molecular property prediction from SMILES, with a large unlabelled SMILES set and small labelled tasks | RoBERTa-style encoder pretrained with multi-task regression on RDKit descriptors, then fine-tuned per task (ChemBERTa-2 MTR) | Lipophilicity RMSE 0.744 for MTR-10M against 1.212 for D-MPNN, author-reported [arxiv:2209.01712 tab. 1]; the 77M corpus is not in the repository [code: chemberta/data/] |

## Implementation sketch

Common core, with the points where the variants differ marked.

```text
corpus  -> tokenizer (SentencePiece in XLNet, CamemBERT and PaLM;
           BLOOM uses a multilingual subword tokenizer [arxiv:2211.05100 §3.3])
           PaLM: lossless, keeps whitespace, digit-split numbers, byte fallback
pack text into fixed-length sequences
           XLNet:    512, [CLS, A, SEP, B, SEP], memory carried across segments
           CamemBERT: 512, whole paragraphs only (one text line each)
           PaLM:     2048, examples concatenated with [eod], no padding
           BLOOM:    2048 tokens per sequence [arxiv:2211.05100 §3.5 table]
           BEHRT:    one visit per sentence, tokens are diagnosis codes;
                     input = code + age + segment + position embeddings
                     [arxiv:1907.09538 §2.3]
           SPECTER:  title + abstract of one paper per input [arxiv:2004.07180 §2.2]
           ChipNeMo: ~9K domain tokens added; new input rows are the mean of
                     the old pieces, output rows zero [arxiv:2311.00176 §2.1]
           BertGCN:  one document node per text, feature = BERT [CLS]
                     [arxiv:2105.05727 §3.1]
           FactualityPrompts: each sentence prefixed "Title ==> " [arxiv:2206.04624 §6.1]
           FPT:      raw data cut into patches, then a trained linear map
                     [arxiv:2103.05247 §2.1, §2.2]
           PPTOD:    prompt + earlier turns -> target text [arxiv:2109.14739 §3.2]
           mGPT:     causal LM, byte-level BPE, 512-token windows
                     [arxiv:2204.07580 §3.2, §3.4]
           DRAGON:   text (up to 512 tokens) + local KG (up to 200 nodes)
                     [arxiv:2210.09338 §3.1, app. B.1]
           OSCAR 22: per-line language ID, then one label per document
                     [arxiv:2201.06642 §4]
           LLaMA:    2048-token sequences; pre-norm RMSNorm, SwiGLU, RoPE
                     [arxiv:2302.13971 §2.2]
           OPT:      GPT-2 byte-level BPE; batch 0.5M to 4M tokens
                     [arxiv:2205.01068 §2.3] [arxiv:2205.01068 tab. 1]
           GLaM:     top-2 expert routing in every other block [arxiv:2112.06905 §4]
           RETRO:    64-token chunks, 2 neighbours each [arxiv:2112.04426 §2.2]
           Pythia:   1024 sequences of 2048 tokens per step [arxiv:2304.01373 §2.4]
           Code Llama: PSM or SPM reordering of prefix, middle, suffix
                     [arxiv:2308.12950 §2.3]
           TaBERT:   question + K rows; each row is a sequence of
                     "column | type | value" cells [arxiv:2005.08314 §3.1]
           BETO:     whole-word masking; length 128, then 512
                     [arxiv:2308.02976 §3]
           phi-1:    2048-token windows; documents joined by end-of-text
                     [arxiv:2306.11644 §2.3]
           C4:       English filter, blocklist, Lorem-ipsum drop, sentence
                     rule [arxiv:2104.08758 §2]
           LinkBERT: anchor segment + second segment (next, random or linked),
                     [CLS] A [SEP] B [SEP] [arxiv:2203.15827 §4.2]
           ProGen2:  terminal tag at each end; sequence used as-is and reversed
                     [arxiv:2206.13517 §3.2]
           CodeGeeX: equal-length segments, each with a language comment tag
                     [arxiv:2303.17568 §2.2]
           BERTje:   sentence-order pairs; paper masks whole words, code does not
                     [arxiv:1912.09582 §2.2] [code: pretraining/create-pretrain-data.py#L32]
           MatSciBERT: full-length 512-token blocks, documents joined by [SEP]
                     [arxiv:2109.15290 §2.2]
           UTC:      UTF-8 bytes, no vocabulary; learned grouping of 1-4 byte blocks,
                     up to 512 bytes [arxiv:2202.11176 §3.1] [§3.4]
           ChemBERTa-2: canonical SMILES, up to 512 tokens [arxiv:2209.01712 §3.1]
for each step:
    build attention mask(s)
           causal LM:      position t sees 1..t
           BLOOM:          as causal LM, plus an ALiBi bias added to the logits
                           (no position embedding) [arxiv:2211.05100 §3.2]
           permutation LM: sample order z; content mask sees z_1..z_t,
                           query mask sees z_1..z_{t-1}; targets = last ~1/K of z
           masked LM:      every position sees the whole sequence
    choose targets
           masked LM:      15% of words, with all subword pieces of a
                           sampled word (CamemBERT)
    forward pass
           permutation LM: two streams with shared weights; logits from the query stream
    GLaM:      softmax gate per token, weighted sum of top-2 experts
               [arxiv:2112.06905 §4]
    RETRO:     retrieve neighbours per chunk; chunked cross-attention to them
               [arxiv:2112.04426 §2.4]
    loss = mean negative log-likelihood over targets
           PaLM: + 1e-4 * log(Z)^2 (z-loss)
    clip gradients, optimizer update
           XLNet: Adam w/ weight decay 0.01, LR 4e-4, 40k warmup, linear decay
           CamemBERT: Adam (beta2 0.98), LR 7e-4, 10k warmup, polynomial decay
           PaLM:  unfactored Adafactor, 1e-2 then 1/sqrt(k), beta2 = 1 - k^-0.8, wd = lr^2
           BLOOM: Adam (0.9, 0.95), LR 6e-5 cosine to 6e-6, wd 0.1, clip 1.0
           ABSA:  AdamW, LR 2e-5, linear decay (code; paper silent) [code: main.py#L171-L177]
           FlauBERT: Adam (0.9, 0.98), wd 0.01, warmup 24k to 6e-4 (base)
                     or 30k to 3e-4 (large); decay not stated [arxiv:1912.05372 §3.2]
           BEHRT:    masked prediction at 15%; optimizer not stated [arxiv:1907.09538 §2.4]
           SPECTER:  triplet margin loss, margin 1, L2 distance; LR 2e-5 and
                     2 epochs (paper) [arxiv:2004.07180 §2.3, §4]
           ChipNeMo: DAPT lr 5e-6, no scheduler, 256 x 4096-token batches
                     [arxiv:2311.00176 §2.2]
           BertGCN:  BERT lr 1e-5, GCN lr 1e-3 [arxiv:2105.05727 §4.1];
                     memory refreshed each epoch [arxiv:2105.05727 §3.3]
           cpt-text: symmetric cross-entropy over in-batch cosine logits,
                     trainable temperature [arxiv:2201.10005 §2.2]
           FactualityPrompts: zero the loss before each sentence's pivot
                     (half-sentence) [arxiv:2206.04624 §6.2]
           FPT:      Adam, lr 1e-3, no schedule; homology lr 1e-4
                     [arxiv:2103.05247 app. C]
           PPTOD:    paper: Adam, no schedule; code: AdamW, lr 5e-5, linear
                     decay [arxiv:2109.14739 §3.4] [code: Pretraining/pretrain.py#L104]
           mGPT:     optimizer and learning rate not stated for final models
                     [arxiv:2204.07580 §3.4]
           DRAGON:   RAdam, lr 2e-5 (LM) and 3e-4 (other), warmup ratio 0.1
                     [arxiv:2210.09338 tab. 7]
           LLaMA:    AdamW (0.9, 0.95), wd 0.1, clip 1.0, 2K warmup, cosine to 10%
                     [arxiv:2302.13971 §2.3]
           OPT:      AdamW (0.9, 0.95), wd 0.1, linear decay to 10% over 300B
                     tokens [arxiv:2205.01068 §2.2]
           GottBERT: peak LR 4e-4 base, 1.5e-4 large; 100k updates, batch 8k
                     [arxiv:2012.02110 §3]
           Code Llama: long-context stage at LR 2e-5, RoPE base 1e6
                     [arxiv:2308.12950 §2.4]
           TaBERT:   LR 2e-5 (K=1) or 4e-5 (K=3), weight decay 0.01, 10 epochs
                     [arxiv:2005.08314 tab. 6]
           BETO:     peak LR 1e-4, 10k warm-up; optimizer not named
                     [arxiv:2308.02976 §3]
           phi-1:    AdamW, peak LR 1e-3, 750 warm-up, weight decay 0.1
                     [arxiv:2306.11644 §2.3]
           LinkBERT: AdamW (0.9, 0.98), weight decay 0.01; tiny peak 5e-3 with
                     5k warm-up, base peak 3e-4 [arxiv:2203.15827 §5.1]
           ProGen2:  Adam (0.9, 0.999), clip 1.0 or 0.8 by size, learning rate
                     per size [arxiv:2206.13517 tab. 1]
           CodeGeeX: Adam, 2000 warm-up steps then cosine decay, peak 1e-4
                     [arxiv:2303.17568 tab. 2]
    PaLM at 540B: on loss spike, rewind ~100 steps and skip 200-500 batches
                  (needs batch = f(step) determinism)
reuse:
    XLNet:     drop query stream, fine-tune per task
    CamemBERT: fine-tune per task with a linear head
    ABSA:      fine-tune the encoder with one task head (linear is enough)
    BLOOM:     prompt, or multitask-finetune (BLOOMZ)
    PaLM:      prompt with 0/1/few examples
    FlauBERT:  fine-tune per task, base or large
    BEHRT:     fine-tune encoder with one linear head per horizon
    SPECTER:   no fine-tuning; use the paper vector as a fixed feature
    ChipNeMo:  SFT on general chat plus ~1.4K domain samples; add RAG
    BertGCN:   predict with the lambda mix of GCN and BERT probabilities
    cpt-text:  linear probe, zero-shot label match, or kNN with k = 256
    WebShop:   imitation of search and choice, then policy gradient
    long-tail: retrieve passages at answer time for rare facts
    FPT:       linear output on the last k tokens; train only the subset
    PPTOD:     prompt with the sub-task, or fine-tune on the target task
    mGPT:      few-shot label likelihood (lowest summed NLL wins)
    DRAGON:    MLP over graph vector, pooled text and interaction node
    BabyLM:    minimal pairs: the acceptable sentence must score higher
    TaBERT:    encoder fine-tuned together with the parser [arxiv:2005.08314 §4.1]
    BETO:      fine-tune per task, cased or uncased per task [arxiv:2308.02976 tab. 1]
    phi-1:     complete a function from its docstring [arxiv:2306.11644 §2.2]
    LinkBERT:  fine-tune per task; MRQA, GLUE and BLURB settings [arxiv:2203.15827 app. B]
    ProGen2:   zero-shot likelihood ranking, or fine-tune on one family [arxiv:2206.13517 §3.3]
    CodeGeeX:  generate or translate with INT8 weights [arxiv:2303.17568 §2.4]
```

Sources: [arxiv:1906.08237 §2.3, §2.5, §3.1, tab. 7] [arxiv:2204.02311 §2, §5, §5.1] [arxiv:1911.03894 §4.2, §4.3] [arxiv:1910.00883 §2.2, §3.1] [arxiv:2211.05100 §3.2, §3.5] [arxiv:1912.05372 §3.2] [arxiv:1907.09538 §2.3, §2.4] [arxiv:2004.07180 §2.3, §4].

## Contradictions

Most cards study different objectives and evaluation regimes and share no benchmark run under the same setup, so they are not compared head to head. The points below are the exceptions where the same kind of number appears in two places.

Data volume is the nearest tension. CamemBERT reports that a 4GB web subset gives almost the same results as 138GB, at a fixed 100k-step BASE budget [arxiv:1911.03894 tab. 5]. PaLM reports that its 62B model was not data-saturated at 780B tokens and kept improving on 1325B tokens [arxiv:2204.02311 app. F]. The regimes differ (small encoder, fixed steps and many passes over a small corpus, against a large decoder and one epoch of a large corpus), and neither card tests the other's setting. Both sides are cited; the page does not resolve it.

The two French encoders disagree on one number. CamemBERT's card reports French XNLI 82.5 for its own model [arxiv:1911.03894 tab. 4]; FlauBERT's table lists CamemBERT at 81.2, cited as reported by CamemBERT's authors [arxiv:1912.05372 tab. 5]. The setups are not the same in the two papers, and neither paper reproduces the other's run, so the page does not resolve it; quote the value of the source you mean.

Within SPECTER, the text gives co-view MAP as 83.8 while Table 1 shows 83.6 [arxiv:2004.07180 §5, tab. 1]; the table is the one to quote.

Within PaLM, some numbers disagree across tables (e.g. "0-shot" HellaSwag 83.6 in the Chinchilla comparison matches the 1-shot entry 83.6 rather than the 0-shot 83.4 of the main table) [arxiv:2204.02311 tab. 4, tab. 21]; quote the table you mean.

Three papers in the newer group contain internal inconsistencies to resolve before quoting. The long-tail paper gives two sizes for the same extrapolation: one quadrillion parameters in its introduction [arxiv:2211.08411 §1] and more than 10^18 in §4.2. This page quotes §4.2 with the fit's caveats. The embedding paper gives a 14.7% gain on Natural Questions in its text, but its Table 5 does not reproduce that figure. By our arithmetic from the Contriever row, the gain is 17.3% at Recall@20 and 6.8% at Recall@100 [arxiv:2201.10005 §1, tab. 5]. The ChipNeMo text gives 23.1 billion tokens for its corpus, while its tables total 24.1 billion for the training blend and 25.3 billion for the raw data [arxiv:2311.00176 §2.2, app. Training Data]. Quote the table value and say which count you use.

Within DRAGON, the average gain is stated three ways. The abstract says about 5% on average, the introduction about 8%, and Table 1 gives about 4.4 points by a simple average computed for this page [arxiv:2210.09338 §1] [arxiv:2210.09338 tab. 1]. Quote the table, not either summary.

Within mGPT, the abstract describes the 13B model as on par with XGLM. Table 7 shows it behind XGLM-7.5B in most few-shot cells [arxiv:2204.07580 tab. 7]. Use the table.

Within BabyLM, the MSGS scores in Table 2 are positive, while the appendix reports mostly negative MCC values for the same models [arxiv:2504.08165 tab. 2] [arxiv:2504.08165 app. D]. Say which column an aggregate uses.

Quality filtering differs across the cards. mGPT uses a classifier that favours Wikipedia-like text [arxiv:2204.07580 §3.1]. OSCAR 22.01 keeps documents and annotates them, and states that its quality is not established [arxiv:2201.06642 §6.2.3]. Neither paper tests the other's filter, so this page does not rank them.

Code decontamination is read two ways. phi-1 concludes that its HumanEval result is not only contamination, because the model retrained on pruned data still beats StarCoder-Prompted [arxiv:2306.11644 §5]. Its own table shows a wide split by overlap: at τ = 0.95, phi-1 scores 81.7% on the 71 problems with a close match in CodeExercises and 26.9% on the other 93 [arxiv:2306.11644 tab. 3]. The size of each group depends on the threshold τ [arxiv:2306.11644 tab. 3]. Separately, the paper's text says phi-1 beats every listed model except GPT-4 on both benchmarks, while its table lists WizardCoder at 57.3% on HumanEval against 50.6% [arxiv:2306.11644 tab. 1]. Quote the table.

Normalization placement is in conflict. GLM-130B reports that Pre-LN, Sandwich-LN and PB-Relax all failed in its FP16 runs, and it uses Post-LN with DeepNorm [arxiv:2210.02414 §3] [app. A]. The LLaMA recipe on this page uses pre-norm RMSNorm [arxiv:2302.13971 §2.2]. The survey adds that one study found pre-norm worse than post-norm for fine-tuned performance [arxiv:2307.06435 §3.8.1]. The runs differ in size, data and precision, so the page does not resolve it.

Positional encoding rankings also conflict. The survey reports one study finding ALiBi better than learned and rotary encodings, and another finding rotary better than ALiBi, and it draws no conclusion [arxiv:2307.06435 §3.8.1]. BLOOM uses ALiBi [arxiv:2211.05100 §3.2], and Galactica tried ALiBi at small scale and kept learned positions [arxiv:2211.09085 §4.1].

## Papers

- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](/papers/1906.08237): permutation LM with two-stream attention on a Transformer-XL backbone; bidirectional context without `[MASK]`, for fine-tuned NLU.
- [PaLM: Scaling Language Modeling with Pathways](/papers/2204.02311): full recipe for a 540B dense causal decoder (architecture, data mix, optimizer, loss-spike handling, pipeline-free parallelism, MFU) and its few-shot results.
- [CamemBERT: a Tasty French Language Model](/papers/1911.03894): RoBERTa-style French encoder with whole-word masking, and ablations on corpus origin, size and masking.
- [FlauBERT: Unsupervised Language Model Pre-training for French](/papers/1912.05372): base and large French BERT-style encoders on a 71 GB cleaned corpus, with the FLUE benchmark; code is non-commercial.
- [BEHRT: Transformer for Electronic Health Records](/papers/1907.09538): BERT over coded diagnosis histories, with masked disease prediction and a head per prediction horizon.
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](/papers/2004.07180): SciBERT continued with a citation triplet loss, giving fixed embeddings for scientific papers.
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](/papers/2211.05100): open 176B causal decoder with ALiBi, ROOTS corpus, full training settings and a paper-vs-code audit.
- [Exploiting BERT for End-to-End Aspect-based Sentiment Analysis](/papers/1910.00883): BERT-base with one task head for joint aspect and sentiment tagging; linear head as strong as richer heads.
- [Self-supervised Learning: Generative or Contrastive](/papers/2006.08218): survey that sorts self-supervised objectives into generative, contrastive and adversarial families; no experiments.
- [Natural Language Processing Advancements By Deep Learning: A Survey](/papers/2003.01200): survey of deep learning across NLP tasks, with pretrained embeddings and task tables; no experiments.
- [A Survey on Aspect-Based Sentiment Analysis: Tasks, Methods, and Challenges](/papers/2203.01054): survey of how pretrained language models are used for aspect-based sentiment tasks; no experiments.
- [BertGCN: Transductive Text Classification by Combining GCN and BERT](/papers/2105.05727): BERT document embeddings as graph nodes, trained jointly with a GCN through a memory bank.
- [Adversarial GLUE](/papers/2111.02840): adversarial test set for five GLUE tasks, built from 14 attack entries, filtered and crowd-checked.
- [Text and Code Embeddings by Contrastive Pre-Training](/papers/2201.10005): contrastive pretraining on paired text and code, giving embeddings for search and classification.
- [Factuality Enhanced Language Models for Open-Ended Text Generation](/papers/2206.04624): FactualityPrompts benchmark, factual-nucleus decoding and title-prefixed sentence-completion training.
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](/papers/2207.01206): simulated shopping environment with BART and BERT components, trained by imitation and then policy gradient.
- [Large Language Models Struggle to Learn Long-Tail Knowledge](/papers/2211.08411): closed-book QA accuracy tracks how often pretraining documents mention both the entities; scale and retrieval compared.
- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](/papers/2305.08283): probes the political lean of 14 models and measures downstream gaps after partisan pretraining.
- [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](/papers/2310.10196): survey and taxonomy of language and foundation models for time series and spatio-temporal data; no experiments.
- [ChipNeMo: Domain-Adapted LLMs for Chip Design](/papers/2311.00176): continued pretraining of LLaMA2 on chip-design text with an extended tokenizer, alignment and retrieval; internal benchmarks.
- [A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model](/papers/2610.00223): end-to-end energy and carbon accounting for a 1.3B to 13B model family, including data, R&D, travel and inference.
- [Pretrained Transformers as Universal Computation Engines](/papers/2103.05247): freezes a language-pretrained GPT-2 and trains only input, output, layer norms and positions, for non-text tasks.
- [Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System](/papers/2109.14739): one T5 model pretrained on partially annotated dialogue corpora, with a text prompt for each sub-task.
- [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](/papers/2201.06642): OSCAR 22.01, a document-level multilingual web corpus with non-destructive quality annotations.
- [mGPT: Few-Shot Learners Go Multilingual](/papers/2204.07580): 1.3B and 13B GPT-3-style models trained on 61 languages, evaluated by few-shot label likelihood.
- [Deep Bidirectional Language-Knowledge Graph Pretraining](/papers/2210.09338): DRAGON, one encoder over text and a local knowledge-graph subgraph, pretrained with MLM and link prediction.
- [Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](/papers/2504.08165): shared word-budget challenge, test suite and meta-analysis of 31 submissions; no new model.
- [LLaMA: Open and Efficient Foundation Language Models](/papers/2302.13971): decoder-only models trained on public data for longer than scaling laws suggest, with a Llama-style block.
- [OPT: Open Pre-trained Transformer Language Models](/papers/2205.01068): open GPT-3-class decoder suite with its training recipe and a published logbook of failures.
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](/papers/2112.06905): sparse decoder with 64 experts and top-2 routing, compared with GPT-3 in-context.
- [Improving language models by retrieving from trillions of tokens](/papers/2112.04426): RETRO, chunked retrieval over a trillion-token database, and RETRO-fitting of frozen models.
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](/papers/2304.01373): eight matched models with public checkpoints, data order and training code, for studying training dynamics.
- [GottBERT: a pure German Language Model](/papers/2012.02110): German-only RoBERTa base and large on raw and filtered German OSCAR.
- [PaLM 2 Technical Report](/papers/2305.10403): compute-optimal scaling, a multilingual mixture of objectives and toxicity control tokens; sizes withheld.
- [Gemma: Open Models Based on Gemini Research and Technology](/papers/2403.08295): 2B and 7B open decoders with pretraining, instruction and RLHF tuning, and benchmark results.
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](/papers/2404.14219): 3.8B model trained on filtered web and synthetic reasoning data, with SFT and DPO and an on-device build.
- [Code Llama: Open Foundation Models for Code](/papers/2308.12950): Llama 2 continued on code, with infilling, long-context fine-tuning and instruction variants.
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](/papers/2101.00027): documented 22-source English corpus used for pretraining and comparison.
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](/papers/2012.13255): fine-tuning in a random low-dimensional subspace; better-pretrained models need fewer dimensions.
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](/papers/2107.13586): survey that frames supervised NLP as prompting a pretrained model; no experiments.
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](/papers/2005.08314): BERT-initialized encoder pretrained on text-table pairs, reading sampled rows with vertical attention; the code is non-commercial.
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](/papers/2104.08758): documentation of C4 by source, contents and removed text; advises against blocklist filtering.
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](/papers/2111.01243): survey of pre-train-then-fine-tune, prompt-based and text-generation uses of pretrained models; no experiments.
- [Textbooks Are All You Need](/papers/2306.11644): phi-1, a 1.3B Python model trained on filtered web code, synthetic textbooks and exercises.
- [Spanish Pre-trained BERT Model and Evaluation Data](/papers/2308.02976): BETO, a Spanish-only BERT-Base with whole-word masking, and the GLUES task suite.
- [Large Language Models: A Survey](/papers/2402.06196): broad survey of how LLMs are built, used and benchmarked; compiled tables from third-party results.
- [LinkBERT: Pretraining Language Models with Document Links](/papers/2203.15827): BERT-style encoder pretrained on linked document pairs with a three-way relation loss; biomedical variant on citation links.
- [ProGen2: Exploring the Boundaries of Protein Language Models](/papers/2206.13517): decoder-only protein language models from 151M to 6.4B parameters, with zero-shot fitness ranking and sequence generation.
- [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](/papers/2303.17568): 13B multilingual code model with the HumanEval-X benchmark and INT8 inference.
- [BERT: A Review of Applications in Natural Language Processing and Understanding](/papers/2103.11943): review of BERT uses in NLP, from domain retraining to attacks; its tables quote the cited papers.
- [Qwen Technical Report](/papers/2309.16609): 1.8B to 14B open decoders on up to 3T tokens, with ChatML SFT, a reward model and PPO, and code and maths variants; results are author-reported.
- [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](/papers/2101.01785): two BERT-Base Arabic encoders, one on formal text and one on tweets, with the ARLUE benchmark.
- [DeBERTaV3](/papers/2111.09543), a DeBERTa variant that uses ELECTRA-style pretraining with gradient-disentangled embedding sharing: replaced token detection with embedding sharing that stops gradients on the generator side.
- [GLM-130B: An Open Bilingual Pre-trained Model](/papers/2210.02414): 130B English and Chinese blank-infilling model, with DeepNorm, FP16 stability tricks and INT4 weight quantization.
- [Galactica: A Large Language Model for Science](/papers/2211.09085): decoder-only models up to 120B trained on a curated scientific corpus, with citation and working-memory tokens.
- [StarCoder: may the source be with you!](/papers/2305.06161): 15.5B code model on permissively licensed code, with multi-query attention and fill-in-the-middle.
- [MotionGPT: Human Motion as a Foreign Language](/papers/2306.14795): motion codes from a VQ-VAE added to a Flan-T5 vocabulary, with span-corruption pretraining and instruction tuning.
- [Can Language Models Learn to Listen?](/papers/2308.10897): GPT-2 Medium predicts 256 VQ listener-motion codes from a speaker's timed transcript; text-only pretraining is ablated.
- [A Comprehensive Overview of Large Language Models](/papers/2307.06435): survey of LLM building blocks, models, tuning, efficiency and evaluation; no experiments.
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](/papers/2406.12793): GLM-4 technical report, with grouped-query attention, long context, alignment and a model that picks its own tools.
- [COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter](/papers/2005.07503): BERT-Large continued on 22.5M COVID-19 tweets with masked-LM and next-sentence losses, then fine-tuned on five classification sets.
- [LongT5: Efficient Text-To-Text Transformer for Long Sequences](/papers/2112.07916): T5 encoder with sliding-window and transient global attention, pretrained with principle-sentence generation, for inputs up to 16k tokens.
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](/papers/2311.16079): Llama-2 continued on a medical corpus of about 47B training tokens with 1% general replay, then fine-tuned per benchmark, with three inference modes.
- [KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media](/papers/2007.13184): language-specific BERT-base with a five-width CNN over its last four layers, for offensive-tweet detection in Arabic, Greek and Turkish; also releases ArabicBERT.
- [Transformer Quality in Linear Time](/papers/2202.10447): FLASH, a gated single-head attention block with exact attention inside chunks and linear attention across them, for linear-cost language modeling.
- [Spam and Sentiment Detection in Arabic Tweets Using MARBERT Model](/papers/2606.25495): MARBERT fine-tuned for five-class sentiment and for spam on private telecom tweets; the rare sarcasm class stays weak.
- [BERTje: A Dutch BERT Model](/papers/1912.09582): BERT-base trained only on Dutch text, with sentence-order prediction and a 12 GB corpus; compared with multilingual BERT.
- [MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction](/papers/2109.15290): continued masked-LM pretraining of a science model on materials full text, with NER and classification heads.
- [A New Generation of Perspective API: Efficient Multilingual Character-level Transformers](/papers/2202.11176): token-free byte-level encoder for multilingual toxicity scoring, with a learned byte-block tokenizer.
- [ChemBERTa-2: Towards Chemical Foundation Models](/papers/2209.01712): RoBERTa-style SMILES models pretrained with masked LM or descriptor regression and compared on MoleculeNet.