Language model pretraining

Overview

Pretraining fits a network to a large corpus of raw data with an objective that needs no labels, so the learned representations or the generator can later be reused on many tasks. Three decisions define a recipe:

1. Objective. A left-to-right (autoregressive, AR) objective maximizes the product of next-token conditionals; each position then only sees one side of its context 1906.08237 §2.1. A denoising autoencoder (AE) objective in the style of BERT (arXiv:1810.04805) reconstructs masked tokens from both sides, but treats the masked tokens as independent of each other and uses a [MASK] symbol that never appears at fine-tuning time 1906.08237 §2.1. A permutation LM keeps the exact AR factorization but over a random order, so every token is trained with context from both sides in expectation 1906.08237 §2.2. A survey files these under generative objectives; ELECTRA's replaced-token detection, which trains a discriminator on a small generator's outputs, is placed in a generative-contrastive family 2006.08218 §2, §5.3. The same survey reports that contrastive pretraining for NLP was unconvincing at its March 2021 version, and that NLP contrastive work mostly sits in fine-tuning 2006.08218 §4.4.
2. How the model is consumed. Either fine-tune the encoder on each task (XLNet's setting, with a separate fine-tuning recipe per task 1906.08237 tab. 8) or prompt a causal decoder with zero, one or a few examples and no gradient updates (PaLM's setting 2204.02311 §1, §6.1). For aspect-based sentiment analysis, later work reformulates the task to suit a pretrained model (sentence-pair classification, reading comprehension, sequence generation) instead of only swapping in its embeddings 2203.01054 §5.
3. Data and scale. Corpus, parameters, tokens, optimizer stability and hardware efficiency. PaLM studies what happens when parameter count is pushed to 540B on one epoch of 780B tokens 2204.02311 §1, §3. CamemBERT studies how much and which kind of web text a single-language encoder needs 1911.03894 §1, §6; FlauBERT trains a larger, cleaned French mix of 71 GB 1912.05372 §3.1.

The invariant across the masked and AR variants: labels come from the data itself, and the training signal is a likelihood or reconstruction loss over items the model did not see. XLNet stresses one part of this explicitly: the permutation lives only in the attention masks, never in the token order 1906.08237 §2.2. SPECTER's labels are citation links between papers, which are also part of the data, and its loss ranks a positive paper above a negative one rather than predicting tokens 2004.07180 §2.3.

The same recipe also serves as a component of larger systems: an encoder that supplies node features to a graph model 2105.05727 §3.1, an embedder trained on paired text 2201.10005 §2.2, and a language-model backbone for a web agent 2207.01206 §4.2. Other cards study what the pretraining data does to the resulting model, including which facts it recalls and which political leanings it carries, and what a training run costs in energy and carbon 2211.08411 §3.1 2305.08283 §4.1 2610.00223 §5.1.

Variants

Permutation language modeling (XLNet)

Sample a factorization order z per sequence and maximize the AR likelihood along z 1906.08237 eq. 3. Because a standard parameterization cannot tell which position it is asked to predict, XLNet uses two-stream self-attention: a content stream (ordinary self-attention including the token) and a query stream that knows the target position but not its token and produces the logits 1906.08237 §2.3, eq. 4. Only the last ≈1/K tokens of z are targets (K = 6 for Large), because the full objective converged slowly 1906.08237 §2.3, tab. 7. The backbone is Transformer-XL with relative positional encodings, segment-level memory and relative segment encodings 1906.08237 §2.4, §2.5. At fine-tuning the query stream is dropped and the model is a normal Transformer-XL 1906.08237 §2.3.

Reported effects (author-reported): with the same data (BooksCorpus + Wikipedia) and BERT's hyperparameters, XLNet-Large beats the best of three BERT-Large variants on all 11 compared benchmarks, e.g. RACE 77.4 vs 75.1 and RTE 81.2 vs 74.0 1906.08237 tab. 1. At Base size, the ablation separates the gains: a DAE objective on a Transformer-XL backbone already beats BERT-Base (RACE 65.03 vs 64.3), and the permutation objective adds more (66.66 with K = 6) 1906.08237 tab. 6. Removing memory hurts most on RACE, the long-context task; adding next-sentence prediction does not reliably help 1906.08237 §3.4, tab. 6. Scaled to 32.89B tokens, XLNet reaches 85.4 on RACE test against RoBERTa's 83.2 1906.08237 §3.1, tab. 2.

Cost: XLNet-Large trained for about 5.5 days on 512 TPU v3 chips (500K steps, batch 8192, length 512) and was still underfitting 1906.08237 §3.1, tab. 7. Several ingredients (span-based target sampling, the "bidirectional data pipeline", memory length) are described only loosely 1906.08237 §3.1.

Card: XLNet: Generalized Autoregressive Pretraining for Language Understanding.

Replaced token detection with shared embeddings (DeBERTaV3)

A small generator fills masked positions, and a discriminator labels every token as original or replaced 2111.09543 §3.1 eq. 3 eq. 4. Unlike masked language modeling, which trains on about 15% of tokens, the discriminator is trained on every token 2111.09543 §2.3.1. The generator has half the discriminator's depth, and the two losses are joined with weight 50 2111.09543 §3.1 §2.3.2. The variant's contribution is how the two models share token embeddings. The discriminator reads the generator's embedding with its gradient stopped and adds a zero-initialised residual that only it trains, which the authors call gradient-disentangled embedding sharing (GDES) 2111.09543 §3.3.

Reported effects (author-reported, 125k-step ablation): GDES reaches 89.3 on MNLI-m against 88.8 for plain sharing and 88.3 for no sharing 2111.09543 tab. 2. On GLUE dev, DeBERTaV3-large averages 91.37, against 90.00 for DeBERTa-large and 89.46 for ELECTRA-large 2111.09543 tab. 3. Caveats: the Section 3 ablations and the Section 4 full runs use different data, batch sizes and step counts 2111.09543 §3.1 §4; the released code defaults to λ = 10 and its scripts use other settings than the paper rtd_task.py; the GLUE and SQuAD numbers are dev-set results with the fine-tuning grid chosen on the same dev sets tab. 11 §6.4. Card: DeBERTaV3.

Dense causal decoder at scale (PaLM)

A decoder-only Transformer trained with mean next-token log-likelihood plus a small z-loss (10^{-4} \cdot \log^2 Z) 2204.02311 §5, in three sizes (8B, 62B, 540B) on the same 780B-token mixture, one epoch, no repeated subcorpus 2204.02311 §3, tab. 1. Architecture changes versus a vanilla Transformer: SwiGLU MLP, parallel attention and MLP in each layer (about 15% faster training at large scale, small quality loss at 8B, none at 62B), multi-query attention, RoPE, tied embeddings, no biases, and a lossless 256k SentencePiece vocabulary that splits numbers into digits 2204.02311 §2.

Training recipe 2204.02311 §5: unfactored Adafactor with parameter scaling, LR 10^{-2} for 10k steps then decayed as 1/\sqrt{k}, \beta_2 = 1 - k^{-0.8}, global-norm clipping 1.0, weight decay lr^{2}, no dropout, 2048-token packed sequences, and a batch ramp from 512 to 2048 sequences for 540B. Data: 50% social-media conversations, 27% quality-weighted web pages, 13% books, 5% GitHub code, 4% Wikipedia, 1% news 2204.02311 tab. 2. The 540B run had about 20 loss spikes; the fix was to rewind about 100 steps and skip 200–500 batches, which relies on a bitwise-deterministic, step-indexed data pipeline 2204.02311 §5, §5.1.

Systems: 2 × 3072 TPU v4 chips, 12-way model × 256-way fully sharded data parallelism inside each pod, 2-way data parallelism across pods, no pipeline parallelism 2204.02311 §4. The paper defines model FLOPs utilization (MFU) and reports 46.2% for PaLM 540B against 21.3–32.5% for GPT-3, Gopher and Megatron-Turing NLG 2204.02311 §4.1, tab. 3.

Reported effects (author-reported): few-shot SOTA on 28 of 29 English NLP tasks 2204.02311 §6.1, tab. 4, 69.3 on 5-shot MMLU against 67.5 for Chinchilla 2204.02311 tab. 6, and 58% on GSM8K with 8-shot chain-of-thought plus a calculator 2204.02311 tab. 10. Gains are not smooth in scale: on about 25% of 150 BIG-bench tasks the 540B score exceeds a log-linear extrapolation from 8B→62B by more than 10 points 2204.02311 §6.2. Training the 62B model on to 1325B tokens raised its MMLU from 53.7 to 62.8, so the 62B model was not data-saturated at 780B tokens 2204.02311 tab. 6, app. F.

Card: PaLM: Scaling Language Modeling with Pathways.

Whole-word masked encoder for one language (CamemBERT)

CamemBERT retrains RoBERTa's recipe on French, with two changes: a SentencePiece tokenizer instead of WordPiece, and whole-word masking 1911.03894 §4.3. The model is BERT-BASE (12 layers, hidden 768, 110M parameters), with a 32k vocabulary learned from 10⁷ sampled sentences 1911.03894 §4.2, §4.3. The objective is MLM with dynamic masking and no next-sentence prediction. Whole-word masking samples 15% of the words and makes every subword piece of a sampled word a target, so no piece of a word can reveal the rest of it 1911.03894 §4.3. Optimization is Adam (β₂ = 0.98), 10k warmup steps to a peak LR of 7e-4, polynomial decay, 100k steps at batch 8192 sequences of at most 512 tokens, on 256 V100 GPUs for about a day 1911.03894 §4.3. The authors note that performance was still rising when they stopped 1911.03894 §4.3.

Reported effects (author-reported, fine-tuned, best of 4 or 10 runs selected on validation): French XNLI 82.5 at 110M parameters, against 80.1 for XLM-R BASE at 270M 1911.03894 tab. 4; UPOS 98.18 and LAS 92.57 on UD GSD 1911.03894 tab. 2. Whole-word masking adds about 0.5 XNLI points and little elsewhere 1911.03894 app. A, tab. 8. Data is the French part of OSCAR (138GB). At equal size, Common Crawl-derived text (OSCAR, CCNet) beat 4GB of Wikipedia, with the largest gaps on the spoken treebank and XNLI 1911.03894 §6.1, tab. 5. The 4GB OSCAR subset came close to the full corpus except on NER, where 138GB was about 0.9 F1 better 1911.03894 §6.2, tab. 5. These comparisons hold at a fixed 100k-step budget, which is about 400 passes over 4GB, and validation loss was still falling 1911.03894 §6.2. The LARGE model, trained on CCNet, gains on NLI and NER but not on tagging or parsing 1911.03894 app. B, tab. 8. Results cover French only 1911.03894 §1.

Card: CamemBERT: a Tasty French Language Model.

Base and large French encoder on a larger mixed corpus (FlauBERT)

FlauBERT is a BERT-style encoder trained with masked language modeling only, in two sizes: base (12 layers, hidden 768) and large (24 layers, hidden 1024) 1912.05372 §3.2, tab. 1. The corpus is 71 GB of cleaned French text, about 12.8 billion tokens, from 24 sub-corpora taken from WMT19, OPUS and Wikimedia 1912.05372 §3.1, tab. 10. Common Crawl is the largest part; it was cut from 215 GB to 43.4 GB by aggressive cleaning that drops very short sentences and repetitive or non-meaningful content 1912.05372 §3.1.

Optimization uses Adam (β₁ = 0.9, β₂ = 0.98, weight decay 0.01) with an effective batch of 8192 sequences. Warmup is 24k steps to a peak of 6e-4 for base, and 30k steps to 3e-4 for large. The paper does not state how the rate decays after warmup 1912.05372 §3.2. The authors report about 410 GPU-hours on 32 V100 GPUs for base and 390 hours on 128 GPUs for large 1912.05372 §3.2; the implied totals of about 13,120 and 49,920 GPU-hours are derived here and not stated in the paper. Large uses pre-norm and the paper describes layer dropping, but that sentence is ambiguous about which sizes it covers 1912.05372 §3.2.

Reported effects (author-reported, fine-tuned on FLUE): both sizes beat multilingual mBERT on most tasks 1912.05372 §5. Large has the top score on book classification (95.00 vs 92.30 for CamemBERT) and on verb sense disambiguation (50.48 F1) 1912.05372 tab. 3, tab. 8. CamemBERT is ahead on PAWS-X paraphrase (90.14 vs 89.49 for FlauBERT-base and 89.34 for large) 1912.05372 tab. 4. On XNLI, FlauBERT reaches 83.4 (large) and 80.6 (base) 1912.05372 tab. 5. The paper's classification head has an extra tanh layer that the authors describe in the text 1912.05372 §5.1; a different head changes the classification numbers, so state which one you use.

Licensing: the paper is CC BY 4.0, but the authors' code repository is licensed for non-commercial use only, and the model weights and the source corpora each need their own check before commercial use 1912.05372 §3.1. Card: FlauBERT: Unsupervised Language Model Pre-training for French.

Monolingual Spanish encoder with whole-word masking (BETO)

BETO is a BERT-Base-sized encoder (12 layers, hidden size 768, 110M parameters) pretrained only on Spanish text, in cased and uncased versions 2308.02976 §3 tab. 1. The corpus is about 3 billion words from Spanish Wikipedia and OPUS sources such as subtitles, news and government journals 2308.02976 §3. Masking is whole-word and dynamic: each sentence gets 10 different masks, and when one sub-word of a word is masked, the other sub-words of that word are masked too. The masking rate is not stated 2308.02976 §3. Pretraining runs 2M steps in two phases: the first 900k at batch size 2048 and length 128, then the rest at batch size 256 and length 512, with peak learning rate 1e-4 after 10k warm-up steps 2308.02976 §3. Fine-tuning follows BERT, with a grid over batch size, learning rate and epochs 2308.02976 §5.1.

Reported effects (author-reported, fine-tuned on each task's Spanish training data): on XNLI the cased model reaches 82.01 against 78.50 for the best multilingual BERT the authors found, and on NER 88.43 against 87.38. Uncased is best on PAWS-X and MLDoc. The paper marks the cased POS score (98.97) and the uncased MLDoc score (96.12) as new state of the art 2308.02976 tab. 1. On question answering the result is mixed: on TAR-to-XQuAD the best multilingual BERT leads on F1 (77.60 against 77.56) 2308.02976 tab. 2. Whether these scores are dev or test is not stated 2308.02976 §5.2. The paper also compiles Spanish tasks into the GLUES suite 2308.02976 §4.

Paper vs code: the linked repository holds a README, a model page, a licence and two replication stubs, but no pretraining or fine-tuning code ner-c.md. Its LICENSE is CC BY 4.0, but the README warns that the authors are not sure all training datasets have licences compatible with CC BY 4.0, "especially for commercial use" README.md. Check each source before commercial use of the weights or the corpus. Card: Spanish Pre-trained BERT Model and Evaluation Data.

Dutch-only BERT with sentence-order prediction (BERTje)

BERTje is a BERT-base encoder pretrained only on Dutch text, about 12 GB or 2.4 billion tokens from books, news, Wikipedia and a reference corpus 1912.09582 §2.1. It replaces next-sentence prediction with sentence-order prediction, where the second segment is either the next sentence or the previous one 1912.09582 §2.2. The paper describes masking 15% of tokens, masking whole words rather than single pieces 1912.09582 §2.2. Fine-tuning runs for four epochs on each task 1912.09582 §4.

Reported effects (author-reported, single runs): on the CoNLL-2002 Dutch NER test set the model scores 88.3 F1, against 80.7 for multilingual BERT 1912.09582 tab. 1. On the 110k Dutch Book Reviews sentiment task it scores 93.0 accuracy, against 89.1 1912.09582 tab. 4. The authors report it ahead of multilingual BERT on every test column of their POS, semantic-role and spatio-temporal tables as well 1912.09582 tab. 2 tab. 3.

Caveats: the released code is research code, and it does not follow the paper's masking. It masks single wordpieces, and it caps predictions at 20 per sequence, so for full-length inputs the masked fraction is far below 15% create-pretrain-data.py [#L34]. The SoNaR split also differs from the paper prepare-sonar.py. Two of the five pretraining sources are not named further in the paper, so the corpus cannot be rebuilt from it 1912.09582 §2.1. Card: BERTje: A Dutch BERT Model.

Arabic encoders with a formal-text and a tweet-trained model (ARBERT and MARBERT)

Both models are BERT-Base encoders with a 100K WordPiece vocabulary, and they differ only in data and objective 2101.01785 §3.1 §3.2. ARBERT is pretrained with whole-word masking on about 61 GB of mostly formal Arabic, and the text gives about 6.5B tokens for it 2101.01785 §3.1 tab. 1. MARBERT is pretrained on 1B tweets sampled from an in-house collection, with no next-sentence objective, because tweets are short 2101.01785 §3.2. Diacritics are removed, and URLs, user mentions and hashtags become placeholder tokens 2101.01785 §3.1 §3.2.

Reported effects (author-reported, fine-tuned on the test splits of the ARLUE benchmark): MARBERT-v2 scores 77.40, above XLM-R Large at 76.55 and ARBERT at 76.07 2101.01785 §5.2. The score averages eight table rows rather than the six task clusters named in the abstract, which is our reading of the table 2101.01785 §5.2. The paper never defines "MARBERT-v2" §5.2. Caveats: the tweet data is not released 2101.01785 §3.2, and the checkpoints are for research use only README.md. Card: ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic.

Fine-tuning MARBERT on imbalanced Arabic tweets (sentiment and spam)

This card reuses MARBERT and does no pretraining of its own, so it shows the fine-tuning side of the recipe. MARBERT is a BERT-style Arabic encoder pretrained on dialectal and standard Arabic tweets, about 15.6B tokens, with the next-sentence objective dropped 2606.25495 §4.2. The authors fine-tune it separately for five-class sentiment and for spam, on private customer tweets from a Saudi telecom company 2606.25495 §5.1. Cleaning removes hashtags, usernames and URLs, collapses noise and duplicate tweets, and maps some letter variants to one form 2606.25495 §4.1. The reported setting is Adam with learning rate 2e-5 and epsilon 1e-8, training batch 96, maximum length 256 and 5 epochs, the best of six rows tried 2606.25495 tab. 4 §6. For the rare classes, the authors rejected over- and undersampling and collected more sarcasm tweets, from about 200 to about 1,400 2606.25495 §5.3.

Reported effects (author-reported, F1): the best sentiment row scores 0.75 2606.25495 tab. 4. Per class, sarcasm rises from 0.33 at training batch 16 to 0.54 at batch 96, and indeterminate from 0.50 to 0.56 2606.25495 tab. 5. The spam model reaches 0.98 on more than 24,000 tweets 2606.25495 tab. 3.

Caveats: no baseline is run on the private data, and the abstract's claim of accuracy gains rests on literature numbers from other datasets 2606.25495 tab. 2. The spam labels come from a keyword list, so the model may learn that list 2606.25495 §5.1. The data is held by the client and no code is released 2606.25495 §5.1. Card: Spam and Sentiment Detection in Arabic Tweets Using MARBERT Model.

Fine-tuned encoder with one light task head (BERT for E2E-ABSA)

The encoder is BERT-base (12 layers, hidden 768, 110M parameters). A single task layer maps its token states to a BIEOS-style tag per word, which marks aspect spans and their sentiment, and the whole stack is fine-tuned 1910.00883 §2, §2.1, §3.3. The head is one of linear, GRU, self-attention, transformer layer or linear-chain CRF 1910.00883 §2.2. Training uses learning rate 2e-5 for 1500 steps, batch 25 (LAPTOP) or 16 (REST), and checkpoint selection on the dev set from step 1000 onward, every 100 steps, averaged over five seeds 1910.00883 §3.1. The paper does not state the optimizer, schedule, clipping, dropout or maximum sequence length; the released code uses AdamW with a linear schedule and clipping at 1.0 main.py main.py, and those are the authors' choices rather than paper facts. Freezing BERT gave results the authors call "far from satisfactory" 1910.00883 §3.4.

Reported effects (author-reported): a linear head already scores 60.43 F1 on LAPTOP and 73.22 on REST, above the best non-BERT baselines at 60.35 and 72.78 1910.00883 tab. 2. The richer heads help little: the best are GRU on LAPTOP (61.12) and self-attention on REST (74.72), i.e. up to 0.7 F1 on LAPTOP and 1.5 F1 on REST above the linear head 1910.00883 tab. 2. The REST setting in the released code concatenates released splits, a combination its README calls invalid, so the REST numbers cannot be checked against released data 1910.00883 §3.1 README.md.

Card: Exploiting BERT for End-to-End Aspect-based Sentiment Analysis.

Language-specific BERT with a CNN over its last four layers (KUISAIL)

One BERT-base encoder per language, with a convolutional classifier on top. For Arabic the encoder is a BERT the authors trained themselves on about 8.2 billion words 2007.13184 §4.3. The classifier takes the outputs of the last four encoder layers as four channels over a 64-token input. It applies 160 filters in five widths (1 to 5 tokens, 32 each), a ReLU and a global max-pool, then a dense sigmoid output for offensive versus not 2007.13184 §4.4 fig. 1. All weights are fine-tuned for 10 epochs at learning rate 2e-5, and the epoch with the best dev macro-F1 is kept 2007.13184 §4.4. The paper does not state the loss, dropout or decision threshold 2007.13184 §4.4.

Reported effects (author-reported, single runs, no variance given): macro-F1 on the OffensEval 2020 subtask A test sets is 0.897 for Arabic, 0.843 for Greek and 0.814 for Turkish, ranked 4th, 4th and 3rd 2007.13184 §1 2007.13184 tab. 2. The average over the three is 0.851, against 0.841 for language-specific BERT with no CNN. Turkish is 0.002 lower with the CNN 2007.13184 tab. 2, a difference computed from the table, not reported by the authors.

Card: KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media.

Open causal decoder with ALiBi (BLOOM)

BLOOM is a 176B-parameter causal decoder-only Transformer trained on ROOTS, a corpus with 46 natural and 13 programming languages 2211.05100 §1, §3.1. Against a vanilla decoder it changes three things: ALiBi attention biases in place of learned or rotary position embeddings, an extra LayerNorm directly after the token embeddings, and tied input and output embeddings 2211.05100 §3.2, §3.5 table. The authors chose the causal objective over encoder-decoder and prefix objectives 2211.05100 §3.2. Shape and optimizer: 70 layers, hidden size 14336, 112 heads, sequence length 2048, global batch 2048 sequences, peak learning rate 6e-5 decaying with a cosine to 6e-6, Adam with (β1, β2) = (0.9, 0.95), weight decay 0.1, gradient clipping at 1.0 and no dropout 2211.05100 §3.5 table 2211.05100 §3.5. Training ran in bfloat16 on 384 A100 80GB GPUs for about 3.5 months 2211.05100 §3.4.

Three caveats the card records. The cosine decay never reaches its end, because warmup plus decay tokens exceed the total token budget 2211.05100 §3.5. The paper's ALiBi slope rule (2^(-8i/n)) is not what the reference code computes when the head count is not a power of two, and 112 is not one 2211.05100 §3.2 transformer.py. The Megatron defaults for Adam β2 and dropout differ from the paper's settings, so set them explicitly arguments.py arguments.py.

Reported effects (author-reported, zero- and few-shot prompting): HumanEval pass@1 of 15.52% for the 176B model 2211.05100 §4.5. SuperGLUE results are shown only in a figure, and the paper reports wide spread across prompts 2211.05100 §4.2, fig. superglue. Multitask finetuning on xP3 (BLOOMZ) significantly increases zero-shot performance over pretrained BLOOM 2211.05100 §4.7.

Card: BLOOM: A 176B-Parameter Open-Access Multilingual Language Model.

Bilingual blank-infilling model at 130B scale (GLM-130B)

GLM-130B is a 130B-parameter English and Chinese model trained on 400B tokens with blank infilling 2210.02414 §1. About 30% of training sequences hold several short spans marked with [MASK], and the rest keep a prefix and mask a tail marked with [gMASK] 2210.02414 §2.2. The architecture has 70 layers, hidden size 12,288 and 96 heads, with DeepNorm Post-LN, one-dimensional RoPE and GeGLU 2210.02414 §2.1 app. A. The stability recipe has three parts: FP16 mixed precision, DeepNorm, and an embedding gradient shrink with factor 0.1 2210.02414 §3. The authors report that Pre-LN, Sandwich-LN and PB-Relax all failed in their runs 2210.02414 §3 app. A.

Reported effects (author-reported): LAMBADA zero-shot accuracy 80.21 against 76.2 for GPT-3 175B, and 44.75 on 5-shot MMLU against 43.9 2210.02414 §5.1 §5.2. Weight-only INT4 quantization with a per-row absolute-maximum scale keeps LAMBADA at 79.47 and MMLU at 44.80, which lets the model run on four 24 GB RTX 3090 cards 2210.02414 §4. Caveats: the training logs record restarts and changes to the shrink factor and learning rate that the one-line summary omits main-log-en.md; the released weights are for non-commercial research MODEL_LICENSE. Card: GLM-130B: An Open Bilingual Pre-trained Model.

Scientific corpus with citation and working tokens (Galactica)

Galactica is a decoder-only family from 125M to 120B parameters, trained on a curated 106B-token corpus, of which 83% is papers 2211.09085 §4.1 tab. 2. Its text is stored as markdown, and special tokens mark citations ([START_REF] and [END_REF], with a title and a surname inside), step-by-step working (<work>), and SMILES, protein and DNA sequences 2211.09085 §3.1 §3.1.1 §3.1.2. Prompt-formatted tasks are included in pre-training instead of added afterwards 2211.09085 §3.2. The models train for 450B tokens, about 4.25 epochs 2211.09085 §5.1.

Reported effects (author-reported, zero-shot): LaTeX equation recall of 68.2% for the 120B model, against 49.0% for GPT-3 2211.09085 tab. 6; 20.4% on MATH with 5-shot mCoT, against 8.8% for PaLM 540B 2211.09085 tab. 9. Caveats: the 120B model was stopped at the start of the fifth epoch, so its final token count is not stated 2211.09085 fig. 6; the authors advise against general-knowledge use 2211.09085 §7.1; the repository gives CC BY-NC 4.0 for the weights LICENSE-MODEL.md. Card: Galactica: A Large Language Model for Science.

Open multilingual GLM family with tool use (ChatGLM)

The ChatGLM report covers the family from GLM-130B to GLM-4 and GLM-4 All Tools 2406.12793 §1. GLM-4 is pretrained on about ten trillion tokens, mostly Chinese and English 2406.12793 §1 §2. Against GLM-130B it drops bias terms except for query, key and value, uses RMSNorm and SwiGLU, extends rotary positions to two dimensions, and uses grouped-query attention with a feed-forward width of 10/3 of the hidden size 2406.12793 §2. Context grows from 2K to 128K through position extension, long-text training and long-context alignment 2406.12793 §2. Alignment combines supervised fine-tuning with RLHF 2406.12793 §2. GLM-4 All Tools is trained to pick a browser, a Python interpreter, a text-to-image model or a user-defined function, and to use each tool's result 2406.12793 §2.

Reported effects (author-reported): GLM-4 (0520) scores 83.3 on MMLU, 93.3 on GSM8K and 78.5 on HumanEval 2406.12793 tab. 2. On the web, All Tools reaches 91.59 on the GSM8K Python task, against 92.72 for GPT-4 (Web) 2406.12793 tab. 9. The report does not state learning rates, batch sizes, step counts or alignment data sizes 2406.12793 §2, so use it to judge relevance rather than to reproduce training. Card: ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools.

Masked model over clinical code histories (BEHRT)

BEHRT applies a BERT-style encoder to diagnosis histories from CPRD, a UK primary-care database, with 301 diagnosis codes after filtering and about 1.6 million patients 1907.09538 §2.2. Each visit is treated as a sentence of codes. A token's input is the sum of four embeddings: the diagnosis, the patient's age, a segment (visit parity) and the visit position 1907.09538 §2.3. Pre-training is masked disease prediction at a 15% masking rate 1907.09538 §2.4. Fine-tuning adds one output layer per task and predicts several diagnoses at once for the next visit, and within 6 and 12 months 1907.09538 §2.4, §2.5. The paper states the architecture sizes and the masking rate, but not the optimizer, learning rate or sequence length 1907.09538 §3, tab. 2.

Reported effects (author-reported): average precision (APS) of 0.462 for the next visit, against 0.360 for DeepR and 0.382 for RETAIN; the same ordering holds at 6 and 12 months 1907.09538 tab. 1. The baselines were tuned with Bayesian optimization, and so were BEHRT's own settings 1907.09538 §3, app. A.

Caveat: the data are licensed through CPRD and the vocabulary is not released, so you must build your own code vocabulary before any reuse 1907.09538 §2.2. Card: BEHRT: Transformer for Electronic Health Records.

Table and text encoder over sampled rows (TaBERT)

TaBERT is a BERT-initialized encoder pretrained on text next to tables: 26.6 million examples in total, 1.3 million from English Wikipedia and 25.3 million from the WDC WebTable Corpus 2005.08314 §3.2 app. A.1. It reads a question together with K rows of the table, not the whole table. Each row becomes one sequence: the question tokens, then one string per cell holding the column name, a data type (text or real) and the cell value 2005.08314 §3.1 eq. 1. A BERT model encodes each row separately, and vertical self-attention then links the same position across rows, with three vertical layers at K = 3 2005.08314 §3.1 §5. At parse time the rows are the K with the highest n-gram overlap with the question; during pretraining they are sampled at random 2005.08314 §3.1 §5. Three losses are summed: masked language modelling on the context, masked column prediction, and cell value recovery 2005.08314 §3.2 app. A.2.

Training uses learning rate 2e-5 for K = 1 and 4e-5 for K = 3, weight decay 0.01 and 10 epochs. The largest model took about 6 days on 128 V100 GPUs 2005.08314 tab. 6 app. A.2.

Reported effects (author-reported): on WikiTableQuestions, MAPO with TaBERT-Large at K = 3 averages 51.8 ± 0.6 on test. The run that was best on dev scores 52.3, above the 46.9 of the previous ten-model ensemble 2005.08314 tab. 1. With K = 1, the test mean is 50.4 for TaBERT-Base and 49.4 for BERT-Base 2005.08314 tab. 1. On Spider dev, TaBERT-Large with K = 3 averages 64.5 ± 0.6, against 61.3 ± 1.2 for BERT-Large with K = 1 2005.08314 tab. 2.

Caveats: the released code is non-commercial (CC BY-NC 4.0), and it does not include the two parsers LICENSE.md 2005.08314 app. B. Its Large K = 3 setup does not run as written, because the hidden size 1024 is not divisible by the default six vertical heads, and the paper gives no head count vertical_attention_table_bert.py. Pick a head count that divides the hidden size, and state it. Card: TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data.

Citation-trained document embeddings (SPECTER)

SPECTER starts from SciBERT and continues training it with a triplet margin loss over citation triples: a query paper, a paper it cites and a paper it does not 2004.07180 §2.3, eq. 2. Distance is L2 with margin 1; the authors report that cosine and dot product were worse 2004.07180 §2.3. The input is the title and abstract only, and the output is one vector per paper, used without fine-tuning 2004.07180 §2.2, §2.5. Training uses about 146K query papers and about 684K triples, five per query (two hard, three easy), from a Semantic Scholar subset 2004.07180 §4. The paper states two epochs, learning rate 2e-5 with a slanted triangular schedule, and an effective batch of 32 built from micro-batches of 4 2004.07180 §4. The authors ran it on one 12 GB GPU, at about one to two days per epoch 2004.07180 §4.

Reported effects (author-reported): average 80.0 over the 12 SciDocs metrics, against 76.9 for SGC, the next-best baseline, and 59.6 for SciBERT with no citation training 2004.07180 tab. 1.

Caveats: the paper's pooling is the [CLS] vector, but the released code averages tokens, and the maximum length differs between the code paths 2004.07180 §2 simple.jsonnet. The paper's learning rate is 2e-5, while the released configuration uses 3e-5 2004.07180 §4 simple.jsonnet. Fix the pooling and length you use and keep them for training and inference. Card: SPECTER: Document-level Representation Learning using Citation-informed Transformers.

Continued domain pretraining with a tokenizer extension (ChipNeMo)

Start from a base model, not a chat-tuned one. LLaMA2 7B, 13B and 70B are each continued with next-token training on a domain corpus 2311.00176 §2.2. Before that, about 9,000 domain tokens are added to the tokenizer. Each new token's input embedding is the mean of the embeddings of the pieces the old tokenizer produced for it, and its output weights start at zero 2311.00176 §2.1, §3.1. The corpus is mostly proprietary chip-design text, with a public slice of Wikipedia and GitHub code that makes up about 9.2% of training tokens 2311.00176 app. Training Data. Training runs for roughly one epoch of the blend at learning rate 5e-6 with no scheduler, using 256 sequences of 4,096 tokens per step 2311.00176 §2.2. Alignment follows: supervised fine-tuning on 128K general chat samples plus 1.4K domain samples, for one epoch 2311.00176 §2.3. Repeating the domain samples for more than one epoch caused overfitting, and running the same continued pretraining on a chat-aligned model degraded its alignment 2311.00176 §2.3.

Reported effects (author-reported, on internal benchmarks): on the design-knowledge multiple-choice set, the 70B model scores 76.6 against 58.4 for GPT-4 and 52.3 for LLaMA2-70B 2311.00176 app. Auto Evaluation Results. On the engineering chatbot with retrieval, it averages 5.68 on a 7-point scale against 4.52 for GPT-4 2311.00176 app. Engineering Assistant Chatbot Human Evaluation. Full continued pretraining clearly beat LoRA adapters (26.4M and 211.2M trainable parameters) on in-domain tasks 2311.00176 §3.4, and the 70B run cost 20,500 A100-hours 2311.00176 §3.5. Retrieval is a separate stage: an e5-small retriever fine-tuned on 3,000 generated samples supplies passages at answer time 2311.00176 §2.4. The domain corpus and the benchmarks are internal, so outsiders cannot rerun them 2311.00176 §3. Card: ChipNeMo: Domain-Adapted LLMs for Chip Design.

Continued pretraining on in-domain tweets (CT-BERT)

CT-BERT starts from BERT-Large (English, uncased, whole-word masking) and keeps training it with masked-token and next-sentence objectives on 22.5M cleaned tweets about COVID-19 2005.07503 §2. Usernames and URLs are replaced with one shared token each, and emojis become text aliases 2005.07503 §2. The paper reports batches of 1024 sequences of at most 96 tokens, run for 500,000 steps at a constant learning rate of 2e-5 2005.07503 §2, §3.1. The released code adds a 10,000-step warmup and linear decay instead, so state which schedule you use run_pretrain.py.

Reported effects (author-reported, dev-split F1 after fine-tuning on five classification sets): the mean is 0.833 against 0.802 for BERT-Large 2005.07503 tab. 2. By relative gain, the COVID-19 category set is largest at 25.88% and SemEval is smallest at 8.97% 2005.07503 tab. 2. One of the five sets, SST-2, is not Twitter data and serves as a control 2005.07503 §2.1.

Caveats: the paper's printed formula for relative gain has the numerator reversed, and the table values match model minus baseline 2005.07503 §3.2. The paper says 30,000 words for the vocabulary, but the config has 30,522 entries bert_config_covid_twitter_bert.json. Fine-tuning epochs and learning rate were tuned on BERT-Large and reused for CT-BERT 2005.07503 §2.1, §4. The 22.5M-tweet corpus is not in the repository 2005.07503 §5, and the MVS dataset is unpublished 2005.07503 §2.1. Card: COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter.

Continued pretraining of Llama-2 on medical text with replay (MEDITRON)

MEDITRON continues Llama-2 at 7B and 70B on GAP-Replay, a medical mixture of about 46.7B training tokens. It holds clinical guidelines from 16 sources, PubMed abstracts, full texts of about 5M open-access PubMed and PMC papers, and 420M replay tokens from RedPajama, about 1% of the mix, to limit forgetting of general text 2311.16079 §2.1 §2.2 §2.3 tab. 1. Inline citations, figure and table references and formulas are replaced by special tokens 2311.16079 §2.2 app. B.2. The architecture and tokenizer are Llama-2's, extended with those tokens 2311.16079 §4.1 app. B.2.

Pretraining uses AdamW (β1 0.9, β2 0.95, ε 1e-5), a cosine schedule with 2000 warmup steps and decay to 10% of the peak, weight decay 0.1 and clipping at 1.0 2311.16079 §4.1. The peak learning rate is 1.5e-4 for 70B and 3e-4 for 7B and 13B, with context 2048 for 7B and 4096 for 70B 2311.16079 §4.1. The 70B run used a global batch of 512, tensor, pipeline and data parallelism of 8, 8 and 2 on 128 A100 80GB GPUs 2311.16079 §3 tab. 1, and took 332 hours 2311.16079 app. A. Each benchmark is then fine-tuned on its own training set for 3 epochs, batch 64, learning rate 2e-5 2311.16079 §4.2. Three inference modes are compared: the top token, chain-of-thought with the zero-shot cue, and self-consistency over 5 samples at temperature 0.8 with a majority vote 2311.16079 §4.3.

Reported effects (author-reported): the 70B model with self-consistency averages 72.0 on the five result columns, against 69.2 for fine-tuned Llama-2-70B in the same mode 2311.16079 tab. 5. In few-shot use without fine-tuning, the 70B average rises from 60.8 at Llama-2 to 63.3 at 23,000 iterations 2311.16079 tab. 6.

Caveats: the paper gives the decay target three ways. The text says 10% of peak, while an appendix study used for pretraining ends near 53% of the peak 2311.16079 §4.1 app. D.1 tab. 11; the paper states no 70B end rate. The data-mixture ablations use the 7B model only 2311.16079 §7.2 tab. 8. Only 8 of 16 guideline sources are released 2311.16079 §2.1. The authors say the models are not ready for clinical use 2311.16079 app. E. Card: MEDITRON-70B: Scaling Medical Pretraining for Large Language Models.

Continued science-model pretraining on materials text (MatSciBERT)

MatSciBERT does not start from scratch. It starts from SciBERT's uncased weights and keeps training with masked language modeling on about 285M words of materials full text and abstracts, from roughly 154,000 papers 2109.15290 §2.1 tab. 1. The recipe is RoBERTa-style: dynamic whole-word masking of 15% of words, no next-sentence loss, and sequences of up to 512 tokens that can span several documents 2109.15290 §2.2. It uses AdamW with a peak learning rate of 1e-4 and linear decay, on two V100 GPUs for about 10 days 2109.15290 §2.2. For fine-tuning, the paper compares a softmax head, a CRF and a two-layer BiLSTM-CRF for entity tagging 2109.15290 §2.4.1. Relation classification marks the two entities with special tokens, and abstract classification uses the CLS output 2109.15290 §2.4.2 §2.4.3.

Reported effects (author-reported, mean of 3 seeds, SciBERT as the only baseline): on SOFC-Slot, the BiLSTM-CRF head reaches Macro-F1 65.92 against 61.68 for SciBERT 2109.15290 tab. 2; the text puts the gap at about 4.2 points §3.3.1. On glass versus non-glass abstracts, F1 is 93.57 against 91.04 tab. 6. On materials-synthesis relation classification the Macro-F1 values are 87.87 and 87.22, and the paper says they are not comparable with prior work tab. 7 §3.3.3.

Caveats: the pretraining corpus is not released. Rebuilding it needs publisher full-text access, labelled abstracts and the family classifiers 2109.15290 §2.1. Only the MatSciBERT path normalises its input text, so part of the gain may come from preprocessing ner.py. The paper describes both a materials vocabulary and SciBERT's vocabulary, and the released code uses SciBERT's 2109.15290 §2.2 §3.2 matscibert_pre_train.py. The repository is GPL-3.0, so only paths and line numbers are cited here. Card: MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction.

Document encoder inside a graph (BertGCN)

The method builds a word-document graph over the whole corpus, in the style of TextGCN. Word-word edges carry positive PMI, and document-word edges carry TF-IDF weights 2105.05727 §3.1, eq. 1. Each document node starts from the BERT or RoBERTa [CLS] embedding, and a two-layer GCN runs over the graph while BERT is trained jointly with it 2105.05727 §3.1, §4.1. Full-batch training would not fit BERT in memory, so a memory bank holds one embedding per document. It is refreshed at the start of each epoch and again for each minibatch, and BERT's learning rate is kept small because the stored rows come from different BERT states 2105.05727 §3.3. The final prediction mixes the GCN output with a BERT-only head, using a weight lambda of 0.7 chosen on a development set 2105.05727 §3.2, §4.3, fig. 1. Author-reported test accuracy with BERT-base is 89.3 on 20NG, 98.1 on R8, 96.6 on R52, 72.8 on Ohsumed and 86.0 on MR. BERT alone scores 85.3, 97.8, 96.4, 70.5 and 85.7 on the same sets 2105.05727 tab. 1. Without the small BERT learning rate, the reported development run did not converge 2105.05727 tab. 2. The released code defaults to RoBERTa and truncates documents at 128 tokens train_bert_gcn.py train_bert_gcn.py. The repository has no license, so this page cites it but does not reuse it. Card: BertGCN: Transductive Text Classification by Combining GCN and BERT.

Contrastive pretraining for text and code embeddings (cpt-text, cpt-code)

Paired data takes the place of labels here: neighbouring passages of web text for the text models, and a docstring with its function for the code models 2201.10005 §1, §3. Each side is encoded separately. The embedding is the last-layer state at an end delimiter, and training pulls true pairs together and pushes them away from every other pair in the batch, with a trainable temperature 2201.10005 §2.1, §2.2. The text models start from GPT-family models and the code models from Codex 2201.10005 §2.2. Batch size matters: on a 300M text model, NQ MRR@10 rises from 71.4 at batch 1536 to 84.7 at 12288 2201.10005 tab. 9. The batch sizes used for the main models shrink as the model grows, and the paper does not give the reason 2201.10005 tab. 1.

Reported effects (author-reported): the largest unsupervised text model, with 175B parameters, reaches 91.8 average linear-probe accuracy on seven SentEval tasks, against 88.3 for DECLUTR 2201.10005 tab. 2. On MSMARCO it reaches MRR@10 of 22.7 against 18.4 for BM25 2201.10005 tab. 5. On CodeSearchNet with 1K candidates, the 1.2B code model averages 93.5 against 77.4 for GraphCodeBERT 2201.10005 tab. 7. Sentence similarity is the weak spot: 71.1 STS average against 79.8 for SimCSE 2201.10005 tab. 4. The paper names its data sources but not their size or filtering, and it has no official code 2201.10005 §3. Card: Text and Code Embeddings by Contrastive Pre-Training.

Continued training against hallucinated facts (FactualityPrompts)

Continued Wikipedia training gets two changes. Each sentence is prefixed with its article title, and the loss ignores tokens before a pivot inside each sentence, by default the middle of the sentence. Decoding stays autoregressive 2206.04624 §6.1, §6.2. At 530B, the vanilla model with top-p decoding has named-entity error of 33.3% on factual prompts and entailment of 11.8%. With both training changes and factual-nucleus decoding, these become 14.5% and 25.5% 2206.04624 tab. 5. Diversity falls from 0.90 to 0.33 in the same comparison 2206.04624 tab. 5. Continued Wikipedia training without the two changes did not help at 1.3B, with named-entity error of 52.5% against 52.4% for the vanilla model 2206.04624 tab. 5. Factual-nucleus decoding shrinks the top-p cut-off as a sentence goes on, resets it at each new sentence and stops it at a floor 2206.04624 §5.1. Most ablations run at 1.3B, and only the best setting is shown at 530B 2206.04624 §5.2, §6.3. The released code covers the metrics and preprocessing, but not the factual-nucleus sampler, and its public checkpoint is smaller than the paper's models README.md. Card: Factuality Enhanced Language Models for Open-Ended Text Generation.

Pretrained backbone for a web agent (WebShop)

In the shopping environment, a BART model turns the instruction into a search string, and a 12-layer BERT scores each clickable button with cross-attention. Both are trained by imitating recorded human sessions, on 1,421 search pairs and 9,558 choice samples 2207.01206 §4.2. The choice model is then tuned with on-line policy gradient while the search model stays frozen, so that generated queries do not drift from language 2207.01206 §4.3. On 500 held-out instructions, imitation alone scores 59.9 with 29.1% success, and imitation plus RL scores 62.4 with 28.7% success. Human experts score 82.1 with 59.6% success, and the rule baseline scores 45.6 with about 10% 2207.01206 tab. 2 2207.01206 §5.2. Removing language pretraining from the choice model cuts success by nearly two-thirds, a result shown only in a figure 2207.01206 §5.2, fig. 4. RL from raw BERT without imitation starts below the rule baseline 2207.01206 §5.2. Card: WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents.

Frozen language-pretrained blocks for other data (FPT)

Keep the self-attention and feedforward blocks of a GPT-2 base model, pretrained on natural language, frozen. Train only a new linear input layer, a new linear output layer, the positional embeddings and the layer-norm parameters 2103.05247 §2.2. The authors apply this to bit strings, MNIST and CIFAR-10 patches, ListOps and protein sequences. The layer norms carry much of the effect: on CIFAR-10, training them alone reaches 54%, against 25% for the output layer alone 2103.05247 tab. 15. Freezing them as well drops 12-layer CIFAR-10 from 68.2% to 61.5% 2103.05247 tab. 16.

Language pretraining beats a random initialisation under the same freeze: 100% against 75.8% on bit memory, and 68.2% against 61.7% on CIFAR-10 2103.05247 tab. 2. The authors also report far fewer gradient steps to convergence, about four times fewer on bit memory and forty times fewer on bit XOR 2103.05247 tab. 6.

Reported effects (author-reported): MNIST reaches 98.0% against 99.1% for a fully trained transformer. CIFAR-10 reaches 72.1% with a 36-layer model, against 70.3% for a fully trained 3-layer GPT-2 2103.05247 tab. 1. Frozen random LSTMs do much worse, for example 16.8% on ListOps against 34.3% for a frozen random transformer 2103.05247 tab. 3 2103.05247 tab. 4.

Caveats: the released code uses an embedding table, not a linear map, for ListOps and homology, reads the bit outputs from the last 20 tokens, clips gradients, and trains every parameter whose name contains "ln" or "norm" 2103.05247 §2.2 experiment.py fpt.py fpt.py. Card: Pretrained Transformers as Universal Computation Engines.

Multi-task text-to-text pretraining for dialogue (PPTOD)

One T5 model handles every sub-task of a task-oriented dialogue system: intent classification, belief-state tracking, dialogue action prediction and response generation. Each sub-task is a text-to-text sample made of a short task prompt, the earlier turns, and the expected output text 2109.14739 §3.2. Pretraining uses eleven human-written dialogue corpora with over 2.3M utterances. Each corpus is annotated for only some sub-tasks, and a sample is made only for the annotations it has 2109.14739 §3.1 2109.14739 tab. 1. Pretraining maximises the likelihood of the target text, and fine-tuning on a target task uses the same objective 2109.14739 §3.2 2109.14739 §3.3.

For end-to-end use, the model predicts the belief state, queries the database with it, and then generates the action and the response from the same database result 2109.14739 §4.1.1. The authors call this a plug-and-play formulation.

Reported effects (author-reported): PPTOD-base has the top MultiWOZ 2.0 combined score in the paper's table, 102.92 2109.14739 tab. 2. With 1% of MultiWOZ 2.0 training data, PPTOD-large reaches joint goal accuracy 31.46, against 13.21 for SOLOIST 2109.14739 tab. 5. The annotation mix matters most at 1%: with no annotation types, end-to-end success is 36.08; with all four, 50.90 2109.14739 tab. 8. A T5-small fine-tuned without multi-task pretraining, with plug-and-play generation and no database state, scores 97.27 combined at 14.17 ms per example. SOLOIST scores 95.74 at 208.69 ms 2109.14739 tab. 6. At 10 samples per intent, intent classification is not the best: USE+ConveRT reaches 85.19 against 84.12 for PPTOD-large 2109.14739 tab. 7.

Caveats: the paper's loss is a sum over target tokens, but the released code averages over the non-padding target tokens 2109.14739 §3.2 dataclass.py. The paper states no fine-tuning optimizer, and the released fine-tuning code uses Adafactor at a much higher learning rate than the Adam setting for pretraining learn.py. Card: Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System.

Long-input text-to-text encoder with transient global attention (LongT5)

LongT5 keeps the T5 encoder-decoder and its text-to-text format, and changes only the encoder's self-attention. Each token attends to a sliding window of radius r = 127 on each side, which adds no parameters 2112.07916 §3.1.1. The TGlobal variant cuts the input into blocks of k = 16 tokens and makes one global token per block, from the summed block embeddings and a normalisation. These globals are rebuilt in every layer and not stored, so no side inputs need designing for each task 2112.07916 §3.1.2. Its state is small: the paper counts about 10k extra parameters for base, 25k for large and 50k for xl 2112.07916 §3.1.2, fn. 1. Earlier long-input models (ETC, BigBird, Longformer) needed such side inputs and used masked-language pretraining 2112.07916 §1, §6. The decoder is unchanged, because the targets are short 2112.07916 §3.1.

Pretraining on C4 uses principle-sentence generation from PEGASUS: sentences are scored by ROUGE-F1 against the rest of the document, the top ones are masked, and the model writes them out as one target. The masked-sentence ratio is 0.2, not PEGASUS's 0.45 2112.07916 §3.2 §4.1.1. In the authors' ablation on NQ and arXiv, this objective beats span corruption, and using both objectives is worse than using this one alone 2112.07916 tab. 5 §5.3.

Reported effects (author-reported, one run per configuration): on PubMed, xl at 16k reaches ROUGE-1 50.23 against 48.36 for HAT-BART, and on BigPatent 76.87 against 60.64 for BigBird-PEGASUS (large) 2112.07916 tab. 2. On TriviaQA, xl at 16k reaches F1 84.83, above BigBird-ETC's 84.5 2112.07916 tab. 4. Multi-News goes to PRIMER and CNN/DM to HAT-BART 2112.07916 tab. 2.

Cost and limits: at input length 2048, LongT5 base TGlobal trains at 765 sequences per second, against 479 for T5.1.1 base, and Local base at 860 2112.07916 §5.1. The longest inputs that fit on the same TPU slice are 6k for T5.1.1 base, 12k for TGlobal base and 36k for Local base 2112.07916 §5.1. Local attention is faster but trails TGlobal and T5.1.1 on NQ 2112.07916 §5.2. Decoding is greedy, and the decoder's full attention means long outputs are not addressed 2112.07916 §4.1 §3.1 §6. Caveats: the paper's comparisons use different input lengths per dataset 2112.07916 tab. 2; one T5.1.1 base entry at 6k has identical EM and F1 (56.73) and may be a transcription error 2112.07916 tab. 4. The released code has the PEGASUS task and data pipeline, but the architecture lives in Flaxformer, which is not pinned to a commit README.md. Card: LongT5: Efficient Text-To-Text Transformer for Long Sequences.

Gated attention with exact attention inside chunks and linear attention across them (FLASH)

A single softmax-free attention head, squared-ReLU weighted, is gated by a GLU-style unit, so the attention itself can be weak 2202.10447 §2, eq. 3, eq. 5. The sequence is cut into fixed-size chunks. Each chunk runs exact attention on itself, and a per-chunk summary carries information across chunks, so cost stays linear in length for a fixed chunk size 2202.10447 §3.2. For causal language modeling the cross-chunk term sums only earlier chunks, so the sequential loop runs over chunks rather than tokens 2202.10447 §3.2, §3.2.1. Pretraining uses 2^18 tokens per batch, 125K steps and a peak learning rate of 7e-4 with AdamW in the reported runs 2202.10447 tab. 6 2202.10447 tab. 7.

Reported effects (author-reported): on Wiki-40B language modeling, the speedup over Transformer++ to reach its quality is 1.2x to 4.9x across contexts from 512 to 8192 2202.10447 §4.2; on C4 masked-LM it is 1.0x to 4.8x 2202.10447 §4.1. At context 8192 on PG-19, perplexity is 41.07 against 43.27 for Transformer+, with a 12.12x speedup against that baseline 2202.10447 tab. 3. These speedups are training-cost ratios, not per-step latency 2202.10447 §4.2. No official code is known; the appendix listings disagree with the equations on scaling and masking, so implementers should test both variants 2202.10447 app. D.

Card: Transformer Quality in Linear Time.

Multilingual causal LM for few-shot use (mGPT)

Two GPT-3-style decoder-only models, about 1.3B and 13B parameters, are trained with next-token prediction on Wikipedia and C4 in 61 languages 2204.07580 §3.1 2204.07580 §3.3 2204.07580 tab. 1. Documents are deduplicated by a 64-bit hash. C4 is filtered, and a Vowpal Wabbit classifier separates Wikipedia documents from the filtered C4 documents as a quality signal 2204.07580 §3.1.

The tokenizer is byte-level BPE with the default strategy, chosen by held-out perplexity over five variants 2204.07580 §3.2 2204.07580 tab. 3. Training uses a 512-token context, a total batch of 2048 and 600k steps, with 400B tokens stated 2204.07580 §3.4. The 1.3B model took 14 days on 256 V100 GPUs and the 13B model 22 days on 512 V100 GPUs 2204.07580 §3.4.

Classification scores each label by the summed negative log-likelihood of its completed prompt, and takes the lowest 2204.07580 §4.2.1. NER and POS tag one word at a time, filling each candidate tag into the prompt 2204.07580 §4.2.2.

Reported effects (author-reported): mGPT-1.3B is close to XGLM-1.7B on classification, with fewer weights and twice the languages 2204.07580 §4.2.1 2204.07580 tab. 7. mGPT-13B beats XGLM-7.5B zero-shot on four of the five classification tasks. In few-shot settings it trails on XWINO and XCOPA at one and four shots, and leads on PAWSX at every shot count and on XNLI at one and four shots 2204.07580 tab. 7. Hate-speech accuracy stays near the 50% chance level 2204.07580 tab. 8. Its 4-shot NER and POS scores are far below fine-tuned encoders such as XLM-R, which the authors note is not a like-for-like comparison 2204.07580 tab. 9 2204.07580 tab. 10 2204.07580 §4.2.2.

Caveats: the released code covers evaluation and inference only. The learning rate, optimizer, schedule, head count and final vocabulary are not stated 2204.07580 §3.4. The tokenizer ablation used a 2048-token context, and the final models 512 2204.07580 §3.2 2204.07580 §3.4. The stated 400B tokens do not match 2048 sequences of 512 tokens over 600k steps, and the paper does not explain the gap. Perplexity is normalised by characters in §3.2 and by tokens in §4.1 2204.07580 eq. 1 2204.07580 §4.1. The code licence is unclear, since the README says MIT and the LICENSE file says otherwise LICENSE.txt. Card: mGPT: Few-Shot Learners Go Multilingual.

Token-free byte-level encoder with a learned tokenizer (UTC)

UTC reads UTF-8 bytes and needs no vocabulary. A learnable tokenizer adapted from Charformer scores blocks of one to four bytes at each position, mixes them with a softmax, and mean-pools the result to half the byte length 2202.11176 §3.1 §4.2. A T5-style encoder-decoder with relative attention biases processes those latent subwords 2202.11176 §3.2 app. A. Pretraining is span denoising on byte spans averaging 20 bytes, covering 15% of the input, with an equal mix of a proprietary Perspective corpus and mC4 2202.11176 §3.4. For classification the decoder is dropped, the encoder output goes through a GeLU layer, the first position is pooled, and a linear layer with sigmoid cross-entropy gives one probability per label 2202.11176 §3.3.

Reported effects (author-reported, proprietary Production-Multilingual set): average AUC-ROC 0.910 for the 102M model and 0.919 for the 268M model, against 0.840 for the in-house multilingual BERT and 0.906 for mT5-base 2202.11176 tab. 1. On the public JMTCC benchmark the 268M model reaches 0.9367, against 0.9239 for mT5-base 2202.11176 tab. 2. On code-switched text the 102M model is below mT5-base (0.9191 against 0.9393), while the 268M model is best 2202.11176 tab. 3.

Caveats: no official code is linked, and the proprietary pretraining corpus means pretraining cannot be repeated exactly 2202.11176 §3.4. The pretraining loss as printed is ambiguous, and the number of labels per task is not stated 2202.11176 §3.3. Card: A New Generation of Perspective API: Efficient Multilingual Character-level Transformers.

Text and knowledge graph in one encoder (DRAGON)

One encoder reads a text segment and a small local knowledge-graph subgraph at once. The two streams exchange information in both directions after each fusion layer 2210.09338 §2.2 2210.09338 fig. 1. The subgraph is built by linking mentions to graph nodes, adding two-hop bridge nodes, and capping the graph at 200 nodes 2210.09338 app. B.1. The encoder has 19 unimodal language-model layers and 5 fusion layers, about 360M parameters in total 2210.09338 §3.1 2210.09338 tab. 7.

Pretraining sums two losses with equal weight: masked language modeling on the text, and link prediction on held-out graph edges scored with DistMult 2210.09338 §2.3 2210.09338 tab. 5. The general model pretrains on BookCorpus with ConceptNet, and the biomedical one on PubMed with UMLS 2210.09338 §3.1 2210.09338 §4.

Reported effects (author-reported): on nine commonsense QA tasks, DRAGON is the best system in every column of 2210.09338 tab. 1. A simple average of its nine columns is about 4.4 points above the continued-MLM RoBERTa baseline. The paper states no average, so this figure is computed for this page 2210.09338 tab. 1. On MedQA it scores 47.5, against 44.6 for BioLinkBERT 2210.09338 tab. 6. In the ablation, MLM alone gives 74.3 on CSQA and link prediction alone 73.8, against 76.0 for both 2210.09338 tab. 5.

Caveats: the link loss as printed has a negative-term sign that leaves it without a finite minimum. The released code uses a bounded form instead 2210.09338 §2.3 2210.09338 eq. 10 modeling_gnn.py. The released pretraining script is a CommonsenseQA demo, not the recipe the paper describes run_pretrain.sh. The authors name generation as future work, since the model encodes and scores but does not write text 2210.09338 §5. Card: Deep Bidirectional Language-Knowledge Graph Pretraining.

Open decoder on public data (LLaMA)

Decoder-only Transformers from 7B to 65B parameters (the paper's table labels them 6.7B to 65.2B). The two smaller models see 1.0T tokens and the two larger ones 1.4T 2302.13971 tab. 2 2302.13971 fig. 1. Each block normalises its inputs with pre-norm RMSNorm, uses SwiGLU feed-forward layers and rotary embeddings, and has no absolute position embeddings 2302.13971 §2.2. The data is public: English CommonCrawl 67%, C4 15%, GitHub 4.5%, Wikipedia 4.5%, Books 4.5%, arXiv 2.5% and Stack Exchange 2%. GitHub is limited to Apache, BSD and MIT projects 2302.13971 §2.1. Training uses AdamW with betas (0.9, 0.95), weight decay 0.1, gradient clipping at 1.0, 2,000 warmup steps and a cosine decay to 10% of the peak rate 2302.13971 §2.3. The reported throughput of about 380 tokens per second per GPU on 2,048 A100-80GB GPUs works out to roughly 21 days for 1.4T tokens 2302.13971 §2.4. The paper does not state dropout, bias terms or weight tying 2302.13971 §2.2.

Reported effects (author-reported): LLaMA-13B beats GPT-3 (175B) on most common-sense benchmarks, and LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B 2302.13971 §3.1 2302.13971 tab. 3. The released code covers inference only 2302.13971 §2.2.

Card: LLaMA: Open and Efficient Foundation Language Models.

Open bilingual decoder with chat alignment and tool use (Qwen)

Decoder-only Transformers in three sizes, 1.8B, 7B and 14B parameters, pretrained on up to 3T tokens. The data mixes public web text, encyclopedias, books and code, with a large English and Chinese share 2309.16609 §2.1 2309.16609 tab. 1. Web text is extracted from HTML, language-identified, deduplicated by exact match and by MinHash with LSH, then filtered with rules and model-based quality scorers 2309.16609 §2.1. The tokenizer is byte-pair encoding starting from cl100k_base, extended with common Chinese characters and words, with a vocabulary of about 152K 2309.16609 §2.2. The architecture uses rotary embeddings, pre-norm RMSNorm and SwiGLU feed-forward layers with width 8/3 of the hidden size; biases appear only in the query, key and value projections, and the input embedding and output projection are untied 2309.16609 §2.3. Pretraining uses AdamW (β₁ 0.9, β₂ 0.95), context 2048, cosine decay to 10% of the peak rate and BFloat16 mixed precision 2309.16609 §2.4.

Long inputs can be handled without training. The paper describes NTK-aware base scaling, dynamic NTK scaling, LogN attention scaling and per-layer window attention, all in words only; it gives no formulas and does not state the window sizes 2309.16609 §2.5. Author-reported perplexity on arXiv papers stays low at 8192 and 16384 tokens with all techniques on 2309.16609 tab. 3.

Alignment has three stages. Supervised fine-tuning uses chat conversations in ChatML format, with system and user turns masked from the loss 2309.16609 §3.1.1 §3.1.2. A reward model is initialised from a same-size model and trained on human comparisons with a pooled end-token reward 2309.16609 §3.2.1. PPO then runs four models at once (policy, value, reference and reward), samples two responses per query, and mixes in a pretraining-data gradient so that benchmark results outside code and maths are not hurt 2309.16609 §3.2.2. Tool use is trained as about 2,000 self-instructed samples in the general SFT mix, using ReAct-style prompts 2309.16609 §3.4. Two specialised models follow: Code-Qwen continues pretraining on about 90B code tokens 2309.16609 §4.1, and Math-Qwen-Chat is fine-tuned directly from the base model on a maths instruction set 2309.16609 §5.1.

Reported effects (author-reported): the 14B base scores 66.3 on MMLU and 72.1 on C-Eval, both 5-shot 2309.16609 tab. 2. The 14B reward model reaches 74.78 pairwise test accuracy on Helpful-base, against 62.68 for the preference-model baseline 2309.16609 tab. 4. The 14B chat model scores 66.5 on MMLU against 83.0 for GPT-4, and the paper says the RLHF model beats the SFT models and trails GPT-4 2309.16609 tab. 5 §3.3.

Card: Qwen Technical Report.

Open decoder with a published training logbook (OPT)

OPT is a suite of decoder-only Transformers from 125M to 175B parameters, trained with a GPT-3-style recipe: AdamW with betas (0.9, 0.95), weight decay 0.1, dropout 0.1, and a learning rate that warms up and then decays linearly to 10% of its peak over 300B tokens 2205.01068 §2.2 2205.01068 tab. 1. The batch is held constant at 0.5M to 4M tokens depending on size 2205.01068 tab. 1. The corpus is about 180B tokens: RoBERTa's data, a subset of the Pile, and Pushshift Reddit threads, with only the longest comment chain kept from each thread and MinHash deduplication 2205.01068 §2.3. The 175B model ran on 992 A100 80GB GPUs with FSDP and Megatron tensor parallelism 2205.01068 §2.4.

Training was unstable. The paper records at least 35 manual restarts and more than 100 hosts cycled out over two months. When the loss diverged, the team lowered the learning rate and restarted from an earlier checkpoint 2205.01068 §2.5. The released launcher uses a lower peak learning rate for 175B than the table in the paper, so the card records both values 2205.01068 tab. 1.

Reported effects (author-reported): OPT-175B is comparable to GPT-3 on the average of standard zero- and few-shot tasks 2205.01068 §3.1 2205.01068 fig. 3. It does worse than GPT-3 on toxicity and CrowS-Pairs 2205.01068 §4.2 2205.01068 §4.4. The stated carbon footprint is 75 tCO2eq, against an estimated 500 tons for GPT-3 2205.01068 §6.

Card: OPT: Open Pre-trained Transformer Language Models.

Sparse mixture-of-experts decoder (GLaM)

GLaM replaces the feed-forward sub-layer of every other block in a decoder-only Transformer with a mixture of 64 experts. Each token is routed to its two highest-scoring experts, and the output is their weighted sum 2112.06905 §4 2112.06905 fig. 2. The gate is a softmax over experts. Whether the two weights are renormalised is not stated 2112.06905 §4. The training loss is next-token cross-entropy plus a GShard-style load-balancing term with weight 0.01 2112.06905 §5.2. The learning rate is constant for the first 10K steps and then decays with the inverse square root of the step count 2112.06905 §5.2. The weights and computation of the large models are sharded with GSPMD on TPU-v4 2112.06905 §4 2112.06905 app. C.

Scale: the 64B/64E model has 1.2T total parameters and 96.6B active per token 2112.06905 tab. 4. Its corpus is 1.6T tokens, with web pages filtered by a classifier that favours curated text and then mixed with books, Wikipedia, forums, news and conversation 2112.06905 §3 2112.06905 tab. 3.

Reported effects (author-reported): GLaM beats GPT-3 (175B) on the average of 29 zero-, one- and few-shot tasks, with about half of GPT-3's inference FLOPs per token and about a third of its training energy 2112.06905 tab. 1.

Card: GLaM: Efficient Scaling of Language Models with Mixture-of-Experts.

Retrieval-enhanced pretraining (RETRO)

RETRO predicts text in 64-token chunks. For each chunk it retrieves neighbour passages from a database of up to about 2 trillion tokens. The keys are time-averaged frozen BERT embeddings of the neighbour text 2112.04426 §2.2 2112.04426 §2.3 2112.04426 §4.1. Neighbours from the training document itself are removed, and neighbours for the training data are fetched in advance 2112.04426 §2.3.

Retrieved passages are encoded by a two-layer bidirectional encoder of width 896 that adds about 19M parameters. They enter through chunked cross-attention in some decoder layers. The first chunk does not depend on retrieval 2112.04426 §2.2 2112.04426 §2.4 2112.04426 §2.5 2112.04426 alg. 1. At sampling time, retrieval runs again after each chunk, so its cost grows with the number of chunks 2112.04426 §2.4.

RETRO-fitting adds the new retrieval weights to a frozen pretrained baseline and trains only those. Without retrieval, the frozen model's output is unchanged 2112.04426 §4.2 2112.04426 app. B.5. Reported effects (author-reported): the 7.5B model outperforms Jurassic-1 and Gopher on a majority of Pile test sets 2112.04426 §4.1 2112.04426 app. F.1. The authors' hyperparameters are partly inconsistent with each other, and no official code was found 2112.04426 app. C.

Card: Improving language models by retrieving from trillions of tokens.

Linked-document pairs for masked pretraining (LinkBERT)

LinkBERT keeps BERT's masked-word objective but changes what one training input contains. An input holds an anchor segment and a second segment. The second segment is the next segment of the same document, a segment from an unrelated document, or a segment from a document the anchor links to, each with probability one third in the main setting 2203.15827 §4.2, §5.1. Linked documents are drawn with probability inversely proportional to their in-degree, so heavily linked pages do not dominate 2203.15827 §4.3. A three-way relation head on the first token replaces BERT's next-sentence prediction, and the loss is the masked-word loss plus the relation loss with equal weight 2203.15827 §4.2 eq. 1. The tiny model starts from random weights; base and large continue from BERT checkpoints 2203.15827 §5.1.

Reported effects (author-reported): average MRQA F1 is 54.1 for LinkBERT-tiny against 50.0 for BERT-tiny, 77.8 against 75.2 for base, and 81.0 against 78.5 for large 2203.15827 tab. 1. The largest gains are on multi-document sets such as HotpotQA, where base rises from 76.0 to 78.2 2203.15827 tab. 1 §5.3. With 10% of the fine-tuning data, base scores 70.5 on HotpotQA against 64.8 for BERT-base 2203.15827 tab. 4. In the biomedical setting, BioLinkBERT-base reaches a BLURB score of 83.39, against 81.10 for PubMedBERT-base at the same size 2203.15827 tab. 7. On tiny, random links match BERT-tiny, and TF-IDF neighbours do better than random but not as well as hyperlinks 2203.15827 §5.5 tab. 5. Removing the relation loss from base costs 1.7 F1 on HotpotQA and 0.8 on SQuAD 2203.15827 tab. 6.

Caveats: the paper does not state the segment length in link-based inputs, how a linked segment is chosen, or what happens to an anchor with no outgoing links 2203.15827 §4.2, §4.3. The released repository holds fine-tuning and preprocessing scripts and models, but not the pretraining code README.md README.md. The MRQA split is a halved dev set, so its numbers are not comparable to the official leaderboard 2203.15827 §5.2. Card: LinkBERT: Pretraining Language Models with Document Links.

Fully documented training suite (Pythia)

Pythia is eight decoder-only models from 70M to 12B parameters. Each is trained on the Pile and on a near-duplicate-removed Pile of about 207B tokens, with the same data order 2304.01373 §2.2 2304.01373 §2.4. The architecture is dense in every layer, uses rotary embeddings, runs attention and the feed-forward sub-layer in parallel, and has untied input and output embeddings 2304.01373 §2.3. Every size uses a global batch of 1024 sequences of 2048 tokens and 143,000 steps, about 300B tokens in total 2304.01373 §2.4 2304.01373 app. B 2304.01373 app. E. The optimizer is Adam with betas (0.9, 0.95) and epsilon 1e-8, and the learning rate decays with a cosine to 0.1 times the peak 2304.01373 app. E 2304.01373 app. B.

The models, checkpoints, training code and data-loader tools are released 2304.01373 §2.7. Use the suite when you need a public model whose training history you can reproduce. Each model has 154 saved checkpoints 2304.01373 §1.

Card: Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling.

German single-language RoBERTa (GottBERT)

GottBERT is a RoBERTa-base or RoBERTa-large encoder trained only on German text. The authors state that it is the first German single-language RoBERTa model 2012.02110 §1 2012.02110 §6. There are two corpora: the German part of OSCAR (145GB), and a 121GB version filtered with rules and a one-class SVM 2012.02110 §3. Base and large are each trained on both corpora, which gives four models, and both the best and last checkpoints are kept 2012.02110 §3 2012.02110 §4.

Pretraining runs 100k updates with batches of 8k sequences, with a peak learning rate of 4e-4 for base and 1.5e-4 for large 2012.02110 §3. Reported effects (author-reported): base models lead on 4 of 6 tasks, and the best filtered and unfiltered checkpoints differ by at most 0.85 points. No seed variance or significance test is reported 2012.02110 tab. 3.

Card: GottBERT: a pure German Language Model.

Compute-optimal multilingual model (PaLM 2)

PaLM 2 is a Transformer in three sizes (S, M and L). Its sizes and architecture are withheld 2305.10403 app. D. The report states three ingredients: compute-optimal scaling, in which data and parameters grow at about the same rate 2305.10403 §1 2305.10403 §2.1; a more multilingual and deduplicated pretraining mixture 2305.10403 §3; and a tuned mixture of objectives motivated by UL2 2305.10403 §1. The scaling study trained at 1e19 to 1e22 FLOPs and chose token counts with FLOPs ≈ 6ND. It fitted a quadratic at each budget and then a power law across budgets 2305.10403 §2.1 2305.10403 app. A.1. Its scaling models (400M to 15B) are not PaLM 2 models 2305.10403 §2.1. The earlier PaLM's non-code data was about 78% English 2305.10403 §1.

Two safety-related pretraining choices: a small fraction of documents carry a toxicity control token from a fixed Perspective API version, and the model can be conditioned on low, medium or high toxicity at inference 2305.10403 §3 2305.10403 §5.1. Synthetic canary sequences measure memorisation 2305.10403 §4. Reported effects (author-reported): BIG-Bench Hard 78.1 with chain-of-thought, against 65.2 for PaLM 2305.10403 tab. 6, and GSM8K 91.0 with self-consistency, against 74.4 for PaLM 2305.10403 tab. 7. Learning rate, batch size and token count are not stated, so the card treats it as a results-and-recipe reference.

Card: PaLM 2 Technical Report.

Small open model with a curated data mix (Gemma)

Gemma 2B and 7B are open decoder-only Transformers, pretrained on 3T and 6T tokens respectively, with instruction-tuned and RLHF-tuned variants 2403.08295 §Pretraining. The authors say the models use the same ideas and training recipes as their larger Gemini models, scaled down 2403.08295 §Pretraining. The technical report gives architecture sizes and benchmark results, but no learning rate or batch size 2403.08295 §Training Infrastructure. Reported effects (author-reported): the 7B pretrained model averages 56.9 over 18 academic benchmarks, against 54.5 for Mistral 7B 2403.08295 tab. 6.

Card: Gemma: Open Models Based on Gemini Research and Technology.

Data-quality-first small model (Phi-3)

phi-3-mini has 3.8B parameters and is trained on 3.3T tokens. Its layout is Llama-2-style, with a 4K default context 2404.14219 §1 2404.14219 §2 2404.14219 tab. 3. Pretraining has two sequential phases: web data for general knowledge, then more heavily filtered web data mixed with synthetic reasoning data 2404.14219 §2. Post-training is supervised fine-tuning followed by DPO 2404.14219 §2. Reported effects (author-reported): results close to Mixtral 8x7B and GPT-3.5 2404.14219 tab. 3. A 4-bit build runs offline on an iPhone 14 at more than 12 tokens per second 2404.14219 §2 2404.14219 fig. 2. No official code was found, and the report leaves core training hyperparameters unstated 2404.14219 §2.

Card: Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone.

Continued code pretraining with infilling (Code Llama)

Code Llama starts from Llama 2 weights and tokenizer and continues next-token training on code: 500B tokens for the 7B, 13B and 34B models, and 1T for 70B 2308.12950 §2.1 2308.12950 §2.6. The mix is 85% code, 8% natural language related to code and 7% general natural language 2308.12950 tab. 1 2308.12950 §2.2. The Python variant adds 100B tokens, 75% of them Python 2308.12950 tab. 1.

Infilling: documents are split into prefix, middle and suffix, then reordered in prefix-suffix-middle or suffix-prefix-middle order, with four special tokens 2308.12950 §2.3. On average it costs 0.6 points at 7B and 1.1 at 13B across HumanEval and MBPP 2308.12950 §3.2. Long-context fine-tuning uses 16,384-token sequences, raises the RoPE base from 10^4 to 10^6 and uses a learning rate of 2e-5 2308.12950 §2.4 2308.12950 §2.6. Only the 70B base model received it 2308.12950 §2.1. Instruction tuning uses about 5B tokens, including about 14,000 self-generated question, test and solution triples checked by unit tests 2308.12950 §2.5. Reported effects (author-reported): HumanEval pass@1 of 33.5 to 53.0 for the base models, and up to 67.8 for Instruct 70B 2308.12950 tab. 2.

Card: Code Llama: Open Foundation Models for Code.

Small code model on filtered and synthetic textbook data (phi-1)

phi-1 is a 1.3B decoder-only Transformer for Python, trained in two stages: pretraining on about 7B tokens of filtered code and synthetic textbooks, then fine-tuning on under 200M tokens of exercises 2306.11644 §2 tab. 1. The filtered code is the Python part of The Stack and StackOverflow. A random forest, trained on GPT-4 ratings of about 100k samples with CodeGen-Mono embeddings as features, scores the pool. The score threshold is not stated 2306.11644 §2.1. GPT-3.5 writes the textbooks under topic and audience constraints, and writes the exercises as docstrings with reference solutions 2306.11644 §2.2. The block design follows CodeGen, with attention and MLP in parallel, rotary embeddings, 24 layers and hidden size 2048 2306.11644 §2.3. Pretraining uses AdamW with peak learning rate 1e-3, 750 warm-up steps and weight decay 0.1, on 1024 sequences of 2048 tokens per step, and the step-24,000 checkpoint is used 2306.11644 §2.3. Fine-tuning uses peak learning rate 1e-4 over 6,000 steps 2306.11644 §2.3.

Reported effects (author-reported): 50.6% pass@1 on HumanEval and 55.5% on MBPP 2306.11644 tab. 1. The same table lists WizardCoder at 57.3% on HumanEval, above phi-1 (see Contradictions). In a 350M ablation, HumanEval pass@1 rises from 12.19% (unfiltered web code, 96k steps) to 17.68% (filtered, 36k steps) and 20.12% with the synthetic textbooks added 2306.11644 §2.1 fig. 1.

Caveats: the checkpoint criterion for fine-tuning is not named 2306.11644 §2.3. The synthetic data comes from GPT-3.5, which the authors say has a high error rate 2306.11644 §6. The prompts are not published 2306.11644 §2.2. The model is Python-only 2306.11644 app. B. Card: Textbooks Are All You Need.

Autoregressive protein language model (ProGen2)

ProGen2 applies the decoder recipe to protein sequences. A left-to-right Transformer predicts the next residue, in sizes from 151M to 6.4B parameters 2206.13517 §3.1 tab. 1. The standard models train on UniRef90 mixed with BFD30 2206.13517 §3.2. A separate antibody model trains on 554M sequences from the Observed Antibody Space, clustered at 85% identity 2206.13517 §3.2. Each sequence gets a terminal tag at each end and is used both as-is and reversed; the weighting between the two orientations is not stated 2206.13517 §3.2. Each block runs attention and the MLP in parallel on one normalised input and uses rotary embeddings 2206.13517 §3.1. Layers, heads, learning rate, batch size, clipping and steps for each size are in Table 1 2206.13517 tab. 1.

Reported effects (author-reported): held-out perplexity on Test-max90 falls from 12.9 to 9.9 as size grows 2206.13517 tab. 2. Zero-shot ranking does not follow size. On the narrow fitness landscapes, base averages Spearman 0.505 and xlarge 0.476 2206.13517 tab. 3 §4.3. On antibody general properties, xlarge scores highest at 0.74 2206.13517 tab. 5.

Caveats: the paper defines perplexity but not the zero-shot score behind its fitness tables. The released code averages forward and reversed log-likelihoods, and the paper does not say whether that produced its tables 2206.13517 §4.1 2206.13517 §4.3 likelihood.py. The release holds inference and scoring code, but no training code or training data README.md. The code's default of 28 layers matches no size in Table 1 2206.13517 tab. 1 configuration_progen.py. Card: ProGen2: Exploring the Boundaries of Protein Language Models.

Molecular SMILES encoder with masked or descriptor pretraining (ChemBERTa-2)

ChemBERTa-2 treats SMILES strings of molecules as text. It is a RoBERTa encoder trained on up to 77M unique PubChem SMILES, in subsets of 5M, 10M and 77M 2209.01712 §3. Two objectives are compared on the same subsets: masked language modeling with 15% of tokens masked 2209.01712 §3.1, and multi-task regression on 200 RDKit descriptors computed from each SMILES, each mean-normalised per property §3.1. The architecture is searched over 50 random configurations of 5M to 46M parameters, and five configurations per objective are trained on the larger subsets 2209.01712 §3.2. Fine-tuning uses scaffold splits of 80/10/10 on MoleculeNet tasks, for up to 100 epochs with early stopping 2209.01712 §3.3.

Reported effects (author-reported, Table 1): the authors find that MTR-pretrained models tend to fine-tune better than MLM-pretrained ones 2209.01712 §4.1. On Lipophilicity RMSE, MTR-10M scores 0.744 against 1.212 for D-MPNN 2209.01712 tab. 1. By the table, ChemBERTa-2 beats D-MPNN on six of eight tasks, and not on BACE classification or ClinTox 2209.01712 §4 tab. 1. Table 1 gives no units and no spread, so RMSE columns should not be compared across tasks tab. 1.

Caveats: the 77M corpus, its canonicalisation and the chosen configurations are not in the repository 2209.01712 §3.2 . The code differs from the paper in places: it divides MTR targets by their standard deviation as well as subtracting the mean roberta_regression.py, caps fine-tuning epochs at 10 and fixes the batch size at 64 finetune.py [#L282-L284]. The repository is MIT-licensed and belongs to a co-author, so it is cited by path. Card: ChemBERTa-2: Towards Chemical Foundation Models.

Motion sequences as words (MotionGPT)

MotionGPT turns human motion into words. A VQ-VAE maps motion to codes from a 512-entry codebook at four-frame downsampling, and the codes join the Flan-T5 vocabulary between start and end tokens 2306.14795 §3.1 §3.2 §4.1. The language model is pretrained with span corruption over text and motion, with about 15% of tokens as noise, then instruction-tuned on prompt templates for text-to-motion, motion-to-text, prediction and in-between 2306.14795 §3.3.

Reported effects (author-reported, HumanML3D): text-to-motion FID 0.232 after task tuning, against 0.116 for T2M-GPT, which is better on this metric 2306.14795 tab. 3. Motion-to-text BLEU@4 is 12.47, against 7.00 for TM2T 2306.14795 tab. 4. In the paper's ablation, instruction tuning lowers text-to-motion FID at all three sizes 2306.14795 tab. 6. Caveats: the paper says the in-between mask is random, but the released code masks one contiguous block 2306.14795 §4.2 mgpt_lm.py; the instruction dataset is promised for release, not released 2306.14795 app. G. Card: MotionGPT: Human Motion as a Foreign Language.

Text-pretrained LM predicting listener gestures (Can Language Models Learn to Listen?)

The MotionGPT pattern applied to listener reactions. Listener facial motion is tokenised by a VQ-VAE into 256 codes. GPT-2 Medium then predicts those codes interleaved with the speaker's timed words, with no audio or video input 2308.10897 §3 app. B. The language model starts from text-only pretrained weights, while the motion tokens get new embeddings 2308.10897 §3.3.

The ablation that isolates initialization differs from the full model only in the pretraining column. On the Trevor Noah split, FD is 22.81 without pretraining and 18.22 with it, and L2 Affect (×10²) is 8.41 against 6.36 2308.10897 tab. 2. Author-reported, single listener. Interleaving and word order also matter in the same table, for example the unaligned variant reaches FD 19.03 2308.10897 tab. 2. The model cannot use cues carried by the speaker's face or prosody, such as sarcasm 2308.10897 §6.4. Card: Can Language Models Learn to Listen?.

Multilingual code model with a benchmark that runs tests (CodeGeeX)

CodeGeeX is a 13B left-to-right Transformer trained on code in 23 languages 2303.17568 §2.1 §2.2. The corpus is 158B tokens, and training runs for 850B tokens over 213,000 steps, which is more than five passes over the corpus 2303.17568 §2.2 §2.3 tab. 2. Each segment gets a language tag written in that language's own comment syntax 2303.17568 §2.2. The network is a 39-layer decoder with hidden size 5120 and 40 heads, plus one extra attention layer on top whose query is the embedding of the next position 2303.17568 §2.1 tab. 2 fig. 2. The vocabulary has 52,224 entries; runs of spaces get their own tokens on top of a byte-pair tokenizer 2303.17568 §2.2. Training uses Adam with 2000 warm-up steps and cosine decay from a peak of 1e-4 2303.17568 §2.3 tab. 2.

The authors also built HumanEval-X: 164 problems hand-translated into C++, Java, JavaScript and Go, scored by running unit tests and reported as pass@k with n = 200 samples per problem 2303.17568 §3.1 §3.2 §4.1. Reported effects (author-reported): average pass@1 is 18.40% on HumanEval-X generation, against 16.73% for CodeGen-Multi-16B 2303.17568 tab. 5. For inference, INT8 weights with absolute-maximum scaling cut the memory for 128-token generation on A100 from 26.9 GB to 14.7 GB 2303.17568 tab. 4 §2.4. The authors report that the INT8 model runs on one RTX 3090, where FP16 runs out of memory 2303.17568 §2.4.

Caveats: the release's pretraining script differs from the paper on learning rate, batch size, sequence length and Adam β2 pretrain_codegeex.sh 2303.17568 tab. 2. The released quantizer takes one maximum per output row, while the paper's equation uses one maximum per layer quantize.py 2303.17568 eq. 4. The weights are licensed for non-commercial research only MODEL_LICENSE. Card: CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X.

Code model on permissively licensed data with fill-in-the-middle (StarCoder)

StarCoderBase is a 15.5B decoder-only code model trained on 1T tokens from The Stack v1.2, which keeps only permissively licensed repositories 2305.06161 §3 §5.5. It reads 8K tokens with multi-query attention, and character-level fill-in-the-middle (FIM) is applied to half of the code files 2305.06161 §5.1 §5.4. Near-duplicates are removed with MinHash over 5-grams at Jaccard 0.7, and any file that holds text from an evaluation set is dropped 2305.06161 §3.5 §5.2. StarCoder is StarCoderBase fine-tuned on about 35B Python tokens 2305.06161 §5.

Reported effects (author-reported, pass@1): HumanEval 33.6 and MBPP 52.7 for StarCoder, against 33.5 and 45.9 for code-cushman-001 2305.06161 tab. 12. StarCoderBase scores 30.4 on HumanEval, below cushman tab. 12. Caveats: the public repository holds a LoRA fine-tuning example, not the pretraining or data code finetune.py; the SPMv2 layout is named but not defined 2305.06161 §5.1; the weights carry an OpenRAIL-M-type licence, not this paper's CC BY-SA 2305.06161 §1. Card: StarCoder: may the source be with you!.

Surveys

A survey of self-supervised learning places methods in three families by objective: generative (encoder plus decoder, reconstruction or likelihood loss), contrastive (encoder plus a small discriminator, similarity loss such as InfoNCE) and generative-contrastive (a generator with a large discriminator) 2006.08218 §2. It treats masked language modeling as a denoising autoencoder and lists its weakness: masked tokens are assumed independent given the unmasked ones 2006.08218 §3.3.3. Use it as a map of the field. Its numbers are second-hand, and it dates from March 2021 2006.08218 fig. 7.

Card: Self-supervised Learning: Generative or Contrastive.

A survey of deep learning for NLP covers pretrained word vectors as the default for rarer words 2003.01200 §3.1.3 and lists BERT among models for named entity recognition 2003.01200 tab. 6. It gives no hyperparameters or code, and its score tables collect numbers from other papers with differing metrics and setups, so do not rank methods from them 2003.01200 tab. 2–7.

Card: Natural Language Processing Advancements By Deep Learning: A Survey.

A survey of large models for time series and spatio-temporal data takes a different angle. It sorts work into a time-series branch and a spatio-temporal branch, and each branch into language-model and foundation-model approaches, then into general and domain-specific work 2310.10196 §3 2310.10196 fig. 3. It has no experiments, and its method table lists category, scope, task and venue for each surveyed method 2310.10196 tab. 2. The authors call the language-model and foundation-model split practical rather than strict, and they note that foundation models for time series are still at an early stage 2310.10196 §3. Card: Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook.

A survey of prompt-based methods rewrites supervised NLP as filling a blank in a pretrained model's input. It has three steps: add the prompt, search for the answer words, and map the answer words to labels 2107.13586 §2.2. Its taxonomy covers prompt engineering, answer engineering, multi-prompt learning and training strategies 2107.13586 §2.3. It reports no numbers of its own 2107.13586 §11.

Card: Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.

A survey of pretrained language models for NLP sorts their use into three paradigms: pre-train then fine-tune, prompt-based learning, and NLP as text generation 2111.01243 §2 §3 §4. For staged fine-tuning it describes a common practice rather than a fixed recipe: keep the pretrained layers frozen while the new layers train, then unfreeze them gradually with a learning rate between 1e-3 and 1e-5 2111.01243 §2.4.2. It runs no experiments of its own 2111.01243 §6. Card: Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey.

A survey of large language models covers how they are built (tokenisers, positional encodings, pretraining objectives, alignment and decoding), how they are used with prompts, retrieval and agents, and the datasets and metrics used to score them 2402.06196 §3 §4 §5. It proposes no method and runs no experiments. Its benchmark tables are compiled from other papers, so compare numbers only within one row 2402.06196 §6.2. Card: Large Language Models: A Survey.

A review of BERT in NLP covers what came after the original model: retraining on domain text, extractive sentence selection, BERTScore, black-box attacks on BERT classifiers, and cross-lingual and domain variants 2103.11943 p. 1. It reports that further training on in-domain text helps, and helps more when the text is on the same subject, but that it can cause forgetting unless the learning rate is low 2103.11943 p. 3. It reports no new experiments. Its tables quote numbers from the papers it cites, such as the SciBERT comparison 2103.11943 p. 11, and its method descriptions are paraphrases, so check numbers and formulas against the originals 2103.11943 p. 6. Card: BERT: A Review of Applications in Natural Language Processing and Understanding.

A survey of large language models with 10B or more parameters covers building blocks, pretrained and tuned models, efficiency methods, datasets, evaluation and applications 2307.06435 §1 §2 §3. It tabulates architecture and optimizer settings for models such as LLaMA, PaLM and BLOOM, and many of its cells are blank or use undefined symbols, so check each row against its source 2307.06435 tab. 5 tab. 6. It runs no experiments of its own, and its top-result table is compiled from cited papers 2307.06435 §5. Card: A Comprehensive Overview of Large Language Models.

Effects of the pretraining corpus

Rare facts

Closed-book question answering accuracy rises with the number of pretraining documents that mention both a question's entity and its answer. The counts come from entity linking over The Pile, ROOTS, C4, OpenWebText and Wikipedia 2211.08411 §2, §3.1. For BLOOM-176B, questions with about 10 relevant documents score 25%, and those with about 10,000 score above 55% 2211.08411 §3.1, fig. 1. Removing the matching C4 documents and retraining a 4.8B model lowers accuracy on those questions. The authors read this as causal, though the text gives no numbers 2211.08411 §3.2, fig. 5. Gains from scale on rare questions are log-linear. A log-linear fit with R² of 0.98 projects more than 10^18 parameters before a supervised baseline or humans would be matched 2211.08411 §4.2, fig. 6. Retrieval helps far more: a BM25 setting that adds the top three Wikipedia paragraphs beats closed-book GPT-Neo at every count, most of all on rare questions 2211.08411 §4.3, fig. 9. The count is only a proxy. In a human check of 300 sampled pairs, about a third of the relevant documents held enough to answer the question 2211.08411 §2.3. GPT-3's counts are estimates scaled from OpenWebText, not measurements 2211.08411 §3. Card: Large Language Models Struggle to Learn Long-Tail Knowledge.

Political lean

Pretrained models carry a political lean, and continued pretraining on partisan text moves it. The authors probe 14 models with a 62-statement political compass test 2305.08283 §2.1, §3. Continued pretraining of RoBERTa-base on reddit-left text moves its social score from 2.97 to -3.03, while most other shifts are small 2305.08283 §4.1, fig. 3. Downstream, RoBERTa variants trained on left or right corpora differ on hate-speech and misinformation detection. Overall F1 gaps reach about 3 points, while the differences between identity groups and news sources are the main finding 2305.08283 tab. 3, tab. 4, §4.2. Vanilla RoBERTa is the top row on several overall metrics 2305.08283 tab. 3. The partisan checkpoints are withheld, and the authors describe training them as a dual-use risk, according to their ethics statement. Card: From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models.

Documented multi-source English corpus (The Pile)

The Pile is a fixed mixture of 22 English sources, about 825 GiB in total. Some sources are given larger weights, so their text is seen more than once in training 2101.00027 §1. The authors report that 1.3B models trained on the Pile beat models trained on CC-100 and raw Common Crawl, on the Pile's own components and on a standard word-prediction benchmark 2101.00027 §1. The paper documents bias, profanity and consent issues for each source, and several sources carry terms-of-service caveats, so check the documentation before reuse 2101.00027 §6.

Card: The Pile: An 800GB Dataset of Diverse Text for Language Modeling.

Document-level web corpus construction (OSCAR 22.01)

OSCAR 22.01 rebuilds a multilingual web corpus from one Common Crawl dump so that each document gets one language label, rather than one label per line 2201.06642 §4. The steps are: trim short lines at the head and tail, drop a document when short lines dominate, identify the language of each line, test whether the document is multilingual, and otherwise keep it under its dominant language if the byte-weighted confidence reaches 0.6 2201.06642 §3 2201.06642 §4 2201.06642 §4.2. Quality is recorded as annotations rather than by deletion: tiny, short_sentences, header, footer, noisy and adult 2201.06642 §5.

Reported effects (author-reported): mid- and high-resource corpus sizes stay close to OSCAR 21.09, while several low-resource corpora shrink 2201.06642 §6.1. The authors do not establish the intrinsic quality of unannotated documents 2201.06642 §6.2.3. In a hand check of the first 100 French documents flagged as adult, 77 were true positives and 23 were false positives, 2 of them on LGBTQI+ sites 2201.06642 §6.2.4.

Caveats: the released code differs from the paper. Its multilingual test needs 10 lines, not 5, and its head and tail trim removes every line before the first long line and after the last one. Its filters also count length in characters in one place and bytes in another multilingual.rs sentence_filter.rs header.rs. Card: Towards a Cleaner Document-Oriented Multilingual Crawled Corpus.

Documenting a web-scraped corpus (C4)

C4 is the cleaned April 2019 Common Crawl snapshot used to train T5. Its English version, C4.en, has 365 million documents, 156 billion tokens and 305 GB 2104.08758 §2 tab. 1. The paper documents three views of it: where the text comes from, what it says, and what the filters removed 2104.08758 §1. Patents.google.com is the most represented website 2104.08758 §3.1. In a sample of 1,000,000 URLs, 92% were first indexed by the Internet Archive between 2011 and 2019, which is an estimate, since the archive date is only a proxy for when a page was written 2104.08758 §3.2.

Benchmark test items occur verbatim in the corpus, after case and punctuation normalisation: from 1.87% to 24.88% of generation targets, and from 1.8% to 53.6% of GLUE-style inputs 2104.08758 §4.2 tab. 4. The bad-word blocklist removes 42% of documents classed as African American English and 32% of Hispanic-aligned ones, against 6.2% and 7.2% for White-aligned and other English 2104.08758 §5.3. The authors advise against blocklist filtering when building a web corpus 2104.08758 §6. A sentiment check finds that "Jewish" co-occurs with more positive words than "Arab" in C4: a 7.5-point gap with one lexicon, and 1.4 and 2.0 points with two others 2104.08758 §4.3 app. A.7.

The paper is a documentation study with no model and no training method. The linked repository holds only a README, with no filter or analysis code and no licence file README.md. The corpus likely contains personal information and copyrighted text, which the authors did not quantify 2104.08758 §6. Use the paper as a checklist for documenting a corpus, not as a licence for the data. Card: Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus.

Evaluation and cost

Robustness under perturbation

Adversarial GLUE is a test set of adversarial examples for five GLUE tasks: SST-2, QQP, MNLI, QNLI and RTE 2111.02840 §3.1. The examples come from 14 attack entries, run against BERT and RoBERTa surrogates, then filtered for transfer and fidelity. Crowd workers check what remains, and an example is kept only when four of five annotators agree with the original label 2111.02840 §3.2, §3.3. Across ten models, including robust-training variants, the average AdvGLUE score ranges from 30.3 to 60.9. On GLUE the same models score 85.7 to 93.2 2111.02840 tab. 4. DeBERTa (Large) is highest on AdvGLUE, and BERT (Large) sits near the bottom. ELECTRA (Large) drops from 93.16 on GLUE to 41.69 2111.02840 tab. 4. Only the dev split is public, and test scores come from a hidden set on CodaLab 2111.02840 §3.5. Card: Adversarial GLUE.

Pretraining under a word budget (BabyLM)

The BabyLM Challenge caps pretraining text at about 100M English words in its Strict and Loose tracks, and at 10M in Strict-Small 2504.08165 §3 2504.08165 tab. 1. Models are scored zero-shot on minimal-pair grammar tests (BLiMP and a hidden supplement), finetuned on a GLUE subset, and finetuned on MSGS 2504.08165 §5.1. The leaderboard aggregate weights grammar at 50%, GLUE at 30% and MSGS at 20% 2504.08165 §5.3. Repeating the same text over several epochs does not count against the budget, so the strongest Strict entry trained for hundreds of epochs 2504.08165 §3 2504.08165 §7.2.

Reported effects (author-reported): the LTG-BERT-based ELC-BERT leads the Strict track with aggregate 0.74, against 0.60 for the best baseline, OPT-125M 2504.08165 tab. 2. In Strict-Small it reaches 0.66, against 0.50 for OPT-125M 2504.08165 tab. 2. Most curriculum-learning entries did not help. Architecture changes, training objectives, data preprocessing and teacher–student distillation gave the larger gains 2504.08165 §7.4.

Caveats: the MSGS column in Table 2 is positive, while the appendix reports mostly negative MCC values for the same models, and the paper does not explain the conversion 2504.08165 tab. 2 2504.08165 app. D. Some submissions go past the word budget through augmentation. The Loose winner builds about 4B words from 100M 2504.08165 §7.2. Card: Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora.

Energy and carbon of a whole project

Pretraining energy is only part of a model project's cost. For Noor, a family of 1.3B to 13B Arabic models trained on 150B tokens each, the authors add data storage and processing, R&D, the four training runs, staff travel and projected inference 2610.00223 §3, §5, §5.1. The project used 59.14 MWh of electricity, and the authors estimate 36.5 t CO2e. Training accounts for 65% of the emissions and flights for 18% 2610.00223 fig. 2, fig. 3. Location matters a great deal: the authors estimate 14.9 t if all the compute had run in France 2610.00223 §7. The inference figures are a prospective estimate that assumes batch size 1 on one A100 2610.00223 §5.1.5. The paper's equivalence between token count and training energy does not follow from its own per-token figure, by our arithmetic in the card 2610.00223 §5.1.5, so use the per-token value instead. Card: A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model.

Pretraining quality and the fine-tuning dimension

Fine-tuning can be reparameterised so that only a small number d of trainable values is learned, with a fixed random projection into the full weight space 2012.13255 §3 2012.13255 eq. 1 2012.13255 eq. 2. For RoBERTa-Large on MRPC, about 200 such values reach 90% of full fine-tuning 2012.13255 §4.2 2012.13255 tab. 1. The smallest d is lower for models pretrained longer 2012.13255 §5.1 2012.13255 fig. 2 and for larger models 2012.13255 §5.2 2012.13255 fig. 3. The authors also link a lower d to better generalisation 2012.13255 §5.3 2012.13255 fig. 4. All numbers are author-reported.

Card: Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning.

When to use which

The cards target different regimes and never compare on a shared setup, so the table below maps constraints to the recipe each card supports rather than ranking them.

Implementation sketch

Common core, with the points where the variants differ marked.

corpus  -> tokenizer (SentencePiece in XLNet, CamemBERT and PaLM;
           BLOOM uses a multilingual subword tokenizer [arxiv:2211.05100 §3.3])
           PaLM: lossless, keeps whitespace, digit-split numbers, byte fallback
pack text into fixed-length sequences
           XLNet:    512, [CLS, A, SEP, B, SEP], memory carried across segments
           CamemBERT: 512, whole paragraphs only (one text line each)
           PaLM:     2048, examples concatenated with [eod], no padding
           BLOOM:    2048 tokens per sequence [arxiv:2211.05100 §3.5 table]
           BEHRT:    one visit per sentence, tokens are diagnosis codes;
                     input = code + age + segment + position embeddings
                     [arxiv:1907.09538 §2.3]
           SPECTER:  title + abstract of one paper per input [arxiv:2004.07180 §2.2]
           ChipNeMo: ~9K domain tokens added; new input rows are the mean of
                     the old pieces, output rows zero [arxiv:2311.00176 §2.1]
           BertGCN:  one document node per text, feature = BERT [CLS]
                     [arxiv:2105.05727 §3.1]
           FactualityPrompts: each sentence prefixed "Title ==> " [arxiv:2206.04624 §6.1]
           FPT:      raw data cut into patches, then a trained linear map
                     [arxiv:2103.05247 §2.1, §2.2]
           PPTOD:    prompt + earlier turns -> target text [arxiv:2109.14739 §3.2]
           mGPT:     causal LM, byte-level BPE, 512-token windows
                     [arxiv:2204.07580 §3.2, §3.4]
           DRAGON:   text (up to 512 tokens) + local KG (up to 200 nodes)
                     [arxiv:2210.09338 §3.1, app. B.1]
           OSCAR 22: per-line language ID, then one label per document
                     [arxiv:2201.06642 §4]
           LLaMA:    2048-token sequences; pre-norm RMSNorm, SwiGLU, RoPE
                     [arxiv:2302.13971 §2.2]
           OPT:      GPT-2 byte-level BPE; batch 0.5M to 4M tokens
                     [arxiv:2205.01068 §2.3] [arxiv:2205.01068 tab. 1]
           GLaM:     top-2 expert routing in every other block [arxiv:2112.06905 §4]
           RETRO:    64-token chunks, 2 neighbours each [arxiv:2112.04426 §2.2]
           Pythia:   1024 sequences of 2048 tokens per step [arxiv:2304.01373 §2.4]
           Code Llama: PSM or SPM reordering of prefix, middle, suffix
                     [arxiv:2308.12950 §2.3]
           TaBERT:   question + K rows; each row is a sequence of
                     "column | type | value" cells [arxiv:2005.08314 §3.1]
           BETO:     whole-word masking; length 128, then 512
                     [arxiv:2308.02976 §3]
           phi-1:    2048-token windows; documents joined by end-of-text
                     [arxiv:2306.11644 §2.3]
           C4:       English filter, blocklist, Lorem-ipsum drop, sentence
                     rule [arxiv:2104.08758 §2]
           LinkBERT: anchor segment + second segment (next, random or linked),
                     [CLS] A [SEP] B [SEP] [arxiv:2203.15827 §4.2]
           ProGen2:  terminal tag at each end; sequence used as-is and reversed
                     [arxiv:2206.13517 §3.2]
           CodeGeeX: equal-length segments, each with a language comment tag
                     [arxiv:2303.17568 §2.2]
           BERTje:   sentence-order pairs; paper masks whole words, code does not
                     [arxiv:1912.09582 §2.2] [code: pretraining/create-pretrain-data.py#L32]
           MatSciBERT: full-length 512-token blocks, documents joined by [SEP]
                     [arxiv:2109.15290 §2.2]
           UTC:      UTF-8 bytes, no vocabulary; learned grouping of 1-4 byte blocks,
                     up to 512 bytes [arxiv:2202.11176 §3.1] [§3.4]
           ChemBERTa-2: canonical SMILES, up to 512 tokens [arxiv:2209.01712 §3.1]
for each step:
    build attention mask(s)
           causal LM:      position t sees 1..t
           BLOOM:          as causal LM, plus an ALiBi bias added to the logits
                           (no position embedding) [arxiv:2211.05100 §3.2]
           permutation LM: sample order z; content mask sees z_1..z_t,
                           query mask sees z_1..z_{t-1}; targets = last ~1/K of z
           masked LM:      every position sees the whole sequence
    choose targets
           masked LM:      15% of words, with all subword pieces of a
                           sampled word (CamemBERT)
    forward pass
           permutation LM: two streams with shared weights; logits from the query stream
    GLaM:      softmax gate per token, weighted sum of top-2 experts
               [arxiv:2112.06905 §4]
    RETRO:     retrieve neighbours per chunk; chunked cross-attention to them
               [arxiv:2112.04426 §2.4]
    loss = mean negative log-likelihood over targets
           PaLM: + 1e-4 * log(Z)^2 (z-loss)
    clip gradients, optimizer update
           XLNet: Adam w/ weight decay 0.01, LR 4e-4, 40k warmup, linear decay
           CamemBERT: Adam (beta2 0.98), LR 7e-4, 10k warmup, polynomial decay
           PaLM:  unfactored Adafactor, 1e-2 then 1/sqrt(k), beta2 = 1 - k^-0.8, wd = lr^2
           BLOOM: Adam (0.9, 0.95), LR 6e-5 cosine to 6e-6, wd 0.1, clip 1.0
           ABSA:  AdamW, LR 2e-5, linear decay (code; paper silent) [code: main.py#L171-L177]
           FlauBERT: Adam (0.9, 0.98), wd 0.01, warmup 24k to 6e-4 (base)
                     or 30k to 3e-4 (large); decay not stated [arxiv:1912.05372 §3.2]
           BEHRT:    masked prediction at 15%; optimizer not stated [arxiv:1907.09538 §2.4]
           SPECTER:  triplet margin loss, margin 1, L2 distance; LR 2e-5 and
                     2 epochs (paper) [arxiv:2004.07180 §2.3, §4]
           ChipNeMo: DAPT lr 5e-6, no scheduler, 256 x 4096-token batches
                     [arxiv:2311.00176 §2.2]
           BertGCN:  BERT lr 1e-5, GCN lr 1e-3 [arxiv:2105.05727 §4.1];
                     memory refreshed each epoch [arxiv:2105.05727 §3.3]
           cpt-text: symmetric cross-entropy over in-batch cosine logits,
                     trainable temperature [arxiv:2201.10005 §2.2]
           FactualityPrompts: zero the loss before each sentence's pivot
                     (half-sentence) [arxiv:2206.04624 §6.2]
           FPT:      Adam, lr 1e-3, no schedule; homology lr 1e-4
                     [arxiv:2103.05247 app. C]
           PPTOD:    paper: Adam, no schedule; code: AdamW, lr 5e-5, linear
                     decay [arxiv:2109.14739 §3.4] [code: Pretraining/pretrain.py#L104]
           mGPT:     optimizer and learning rate not stated for final models
                     [arxiv:2204.07580 §3.4]
           DRAGON:   RAdam, lr 2e-5 (LM) and 3e-4 (other), warmup ratio 0.1
                     [arxiv:2210.09338 tab. 7]
           LLaMA:    AdamW (0.9, 0.95), wd 0.1, clip 1.0, 2K warmup, cosine to 10%
                     [arxiv:2302.13971 §2.3]
           OPT:      AdamW (0.9, 0.95), wd 0.1, linear decay to 10% over 300B
                     tokens [arxiv:2205.01068 §2.2]
           GottBERT: peak LR 4e-4 base, 1.5e-4 large; 100k updates, batch 8k
                     [arxiv:2012.02110 §3]
           Code Llama: long-context stage at LR 2e-5, RoPE base 1e6
                     [arxiv:2308.12950 §2.4]
           TaBERT:   LR 2e-5 (K=1) or 4e-5 (K=3), weight decay 0.01, 10 epochs
                     [arxiv:2005.08314 tab. 6]
           BETO:     peak LR 1e-4, 10k warm-up; optimizer not named
                     [arxiv:2308.02976 §3]
           phi-1:    AdamW, peak LR 1e-3, 750 warm-up, weight decay 0.1
                     [arxiv:2306.11644 §2.3]
           LinkBERT: AdamW (0.9, 0.98), weight decay 0.01; tiny peak 5e-3 with
                     5k warm-up, base peak 3e-4 [arxiv:2203.15827 §5.1]
           ProGen2:  Adam (0.9, 0.999), clip 1.0 or 0.8 by size, learning rate
                     per size [arxiv:2206.13517 tab. 1]
           CodeGeeX: Adam, 2000 warm-up steps then cosine decay, peak 1e-4
                     [arxiv:2303.17568 tab. 2]
    PaLM at 540B: on loss spike, rewind ~100 steps and skip 200-500 batches
                  (needs batch = f(step) determinism)
reuse:
    XLNet:     drop query stream, fine-tune per task
    CamemBERT: fine-tune per task with a linear head
    ABSA:      fine-tune the encoder with one task head (linear is enough)
    BLOOM:     prompt, or multitask-finetune (BLOOMZ)
    PaLM:      prompt with 0/1/few examples
    FlauBERT:  fine-tune per task, base or large
    BEHRT:     fine-tune encoder with one linear head per horizon
    SPECTER:   no fine-tuning; use the paper vector as a fixed feature
    ChipNeMo:  SFT on general chat plus ~1.4K domain samples; add RAG
    BertGCN:   predict with the lambda mix of GCN and BERT probabilities
    cpt-text:  linear probe, zero-shot label match, or kNN with k = 256
    WebShop:   imitation of search and choice, then policy gradient
    long-tail: retrieve passages at answer time for rare facts
    FPT:       linear output on the last k tokens; train only the subset
    PPTOD:     prompt with the sub-task, or fine-tune on the target task
    mGPT:      few-shot label likelihood (lowest summed NLL wins)
    DRAGON:    MLP over graph vector, pooled text and interaction node
    BabyLM:    minimal pairs: the acceptable sentence must score higher
    TaBERT:    encoder fine-tuned together with the parser [arxiv:2005.08314 §4.1]
    BETO:      fine-tune per task, cased or uncased per task [arxiv:2308.02976 tab. 1]
    phi-1:     complete a function from its docstring [arxiv:2306.11644 §2.2]
    LinkBERT:  fine-tune per task; MRQA, GLUE and BLURB settings [arxiv:2203.15827 app. B]
    ProGen2:   zero-shot likelihood ranking, or fine-tune on one family [arxiv:2206.13517 §3.3]
    CodeGeeX:  generate or translate with INT8 weights [arxiv:2303.17568 §2.4]

Sources: 1906.08237 §2.3, §2.5, §3.1, tab. 7 2204.02311 §2, §5, §5.1 1911.03894 §4.2, §4.3 1910.00883 §2.2, §3.1 2211.05100 §3.2, §3.5 1912.05372 §3.2 1907.09538 §2.3, §2.4 2004.07180 §2.3, §4.

Contradictions

Most cards study different objectives and evaluation regimes and share no benchmark run under the same setup, so they are not compared head to head. The points below are the exceptions where the same kind of number appears in two places.

Data volume is the nearest tension. CamemBERT reports that a 4GB web subset gives almost the same results as 138GB, at a fixed 100k-step BASE budget 1911.03894 tab. 5. PaLM reports that its 62B model was not data-saturated at 780B tokens and kept improving on 1325B tokens 2204.02311 app. F. The regimes differ (small encoder, fixed steps and many passes over a small corpus, against a large decoder and one epoch of a large corpus), and neither card tests the other's setting. Both sides are cited; the page does not resolve it.

The two French encoders disagree on one number. CamemBERT's card reports French XNLI 82.5 for its own model 1911.03894 tab. 4; FlauBERT's table lists CamemBERT at 81.2, cited as reported by CamemBERT's authors 1912.05372 tab. 5. The setups are not the same in the two papers, and neither paper reproduces the other's run, so the page does not resolve it; quote the value of the source you mean.

Within SPECTER, the text gives co-view MAP as 83.8 while Table 1 shows 83.6 2004.07180 §5, tab. 1; the table is the one to quote.

Within PaLM, some numbers disagree across tables (e.g. "0-shot" HellaSwag 83.6 in the Chinchilla comparison matches the 1-shot entry 83.6 rather than the 0-shot 83.4 of the main table) 2204.02311 tab. 4, tab. 21; quote the table you mean.

Three papers in the newer group contain internal inconsistencies to resolve before quoting. The long-tail paper gives two sizes for the same extrapolation: one quadrillion parameters in its introduction 2211.08411 §1 and more than 10^18 in §4.2. This page quotes §4.2 with the fit's caveats. The embedding paper gives a 14.7% gain on Natural Questions in its text, but its Table 5 does not reproduce that figure. By our arithmetic from the Contriever row, the gain is 17.3% at Recall@20 and 6.8% at Recall@100 2201.10005 §1, tab. 5. The ChipNeMo text gives 23.1 billion tokens for its corpus, while its tables total 24.1 billion for the training blend and 25.3 billion for the raw data 2311.00176 §2.2, app. Training Data. Quote the table value and say which count you use.

Within DRAGON, the average gain is stated three ways. The abstract says about 5% on average, the introduction about 8%, and Table 1 gives about 4.4 points by a simple average computed for this page 2210.09338 §1 2210.09338 tab. 1. Quote the table, not either summary.

Within mGPT, the abstract describes the 13B model as on par with XGLM. Table 7 shows it behind XGLM-7.5B in most few-shot cells 2204.07580 tab. 7. Use the table.

Within BabyLM, the MSGS scores in Table 2 are positive, while the appendix reports mostly negative MCC values for the same models 2504.08165 tab. 2 2504.08165 app. D. Say which column an aggregate uses.

Quality filtering differs across the cards. mGPT uses a classifier that favours Wikipedia-like text 2204.07580 §3.1. OSCAR 22.01 keeps documents and annotates them, and states that its quality is not established 2201.06642 §6.2.3. Neither paper tests the other's filter, so this page does not rank them.

Code decontamination is read two ways. phi-1 concludes that its HumanEval result is not only contamination, because the model retrained on pruned data still beats StarCoder-Prompted 2306.11644 §5. Its own table shows a wide split by overlap: at τ = 0.95, phi-1 scores 81.7% on the 71 problems with a close match in CodeExercises and 26.9% on the other 93 2306.11644 tab. 3. The size of each group depends on the threshold τ 2306.11644 tab. 3. Separately, the paper's text says phi-1 beats every listed model except GPT-4 on both benchmarks, while its table lists WizardCoder at 57.3% on HumanEval against 50.6% 2306.11644 tab. 1. Quote the table.

Normalization placement is in conflict. GLM-130B reports that Pre-LN, Sandwich-LN and PB-Relax all failed in its FP16 runs, and it uses Post-LN with DeepNorm 2210.02414 §3 app. A. The LLaMA recipe on this page uses pre-norm RMSNorm 2302.13971 §2.2. The survey adds that one study found pre-norm worse than post-norm for fine-tuned performance 2307.06435 §3.8.1. The runs differ in size, data and precision, so the page does not resolve it.

Positional encoding rankings also conflict. The survey reports one study finding ALiBi better than learned and rotary encodings, and another finding rotary better than ALiBi, and it draws no conclusion 2307.06435 §3.8.1. BLOOM uses ALiBi 2211.05100 §3.2, and Galactica tried ALiBi at small scale and kept learned positions 2211.09085 §4.1.

Papers

• XLNet: Generalized Autoregressive Pretraining for Language Understanding: permutation LM with two-stream attention on a Transformer-XL backbone; bidirectional context without [MASK], for fine-tuned NLU.
• PaLM: Scaling Language Modeling with Pathways: full recipe for a 540B dense causal decoder (architecture, data mix, optimizer, loss-spike handling, pipeline-free parallelism, MFU) and its few-shot results.
• CamemBERT: a Tasty French Language Model: RoBERTa-style French encoder with whole-word masking, and ablations on corpus origin, size and masking.
• FlauBERT: Unsupervised Language Model Pre-training for French: base and large French BERT-style encoders on a 71 GB cleaned corpus, with the FLUE benchmark; code is non-commercial.
• BEHRT: Transformer for Electronic Health Records: BERT over coded diagnosis histories, with masked disease prediction and a head per prediction horizon.
• SPECTER: Document-level Representation Learning using Citation-informed Transformers: SciBERT continued with a citation triplet loss, giving fixed embeddings for scientific papers.
• BLOOM: A 176B-Parameter Open-Access Multilingual Language Model: open 176B causal decoder with ALiBi, ROOTS corpus, full training settings and a paper-vs-code audit.
• Exploiting BERT for End-to-End Aspect-based Sentiment Analysis: BERT-base with one task head for joint aspect and sentiment tagging; linear head as strong as richer heads.
• Self-supervised Learning: Generative or Contrastive: survey that sorts self-supervised objectives into generative, contrastive and adversarial families; no experiments.
• Natural Language Processing Advancements By Deep Learning: A Survey: survey of deep learning across NLP tasks, with pretrained embeddings and task tables; no experiments.
• A Survey on Aspect-Based Sentiment Analysis: Tasks, Methods, and Challenges: survey of how pretrained language models are used for aspect-based sentiment tasks; no experiments.
• BertGCN: Transductive Text Classification by Combining GCN and BERT: BERT document embeddings as graph nodes, trained jointly with a GCN through a memory bank.
• Adversarial GLUE: adversarial test set for five GLUE tasks, built from 14 attack entries, filtered and crowd-checked.
• Text and Code Embeddings by Contrastive Pre-Training: contrastive pretraining on paired text and code, giving embeddings for search and classification.
• Factuality Enhanced Language Models for Open-Ended Text Generation: FactualityPrompts benchmark, factual-nucleus decoding and title-prefixed sentence-completion training.
• WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents: simulated shopping environment with BART and BERT components, trained by imitation and then policy gradient.
• Large Language Models Struggle to Learn Long-Tail Knowledge: closed-book QA accuracy tracks how often pretraining documents mention both the entities; scale and retrieval compared.
• From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models: probes the political lean of 14 models and measures downstream gaps after partisan pretraining.
• Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook: survey and taxonomy of language and foundation models for time series and spatio-temporal data; no experiments.
• ChipNeMo: Domain-Adapted LLMs for Chip Design: continued pretraining of LLaMA2 on chip-design text with an extended tokenizer, alignment and retrieval; internal benchmarks.
• A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model: end-to-end energy and carbon accounting for a 1.3B to 13B model family, including data, R&D, travel and inference.
• Pretrained Transformers as Universal Computation Engines: freezes a language-pretrained GPT-2 and trains only input, output, layer norms and positions, for non-text tasks.
• Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System: one T5 model pretrained on partially annotated dialogue corpora, with a text prompt for each sub-task.
• Towards a Cleaner Document-Oriented Multilingual Crawled Corpus: OSCAR 22.01, a document-level multilingual web corpus with non-destructive quality annotations.
• mGPT: Few-Shot Learners Go Multilingual: 1.3B and 13B GPT-3-style models trained on 61 languages, evaluated by few-shot label likelihood.
• Deep Bidirectional Language-Knowledge Graph Pretraining: DRAGON, one encoder over text and a local knowledge-graph subgraph, pretrained with MLM and link prediction.
• Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora: shared word-budget challenge, test suite and meta-analysis of 31 submissions; no new model.
• LLaMA: Open and Efficient Foundation Language Models: decoder-only models trained on public data for longer than scaling laws suggest, with a Llama-style block.
• OPT: Open Pre-trained Transformer Language Models: open GPT-3-class decoder suite with its training recipe and a published logbook of failures.
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts: sparse decoder with 64 experts and top-2 routing, compared with GPT-3 in-context.
• Improving language models by retrieving from trillions of tokens: RETRO, chunked retrieval over a trillion-token database, and RETRO-fitting of frozen models.
• Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling: eight matched models with public checkpoints, data order and training code, for studying training dynamics.
• GottBERT: a pure German Language Model: German-only RoBERTa base and large on raw and filtered German OSCAR.
• PaLM 2 Technical Report: compute-optimal scaling, a multilingual mixture of objectives and toxicity control tokens; sizes withheld.
• Gemma: Open Models Based on Gemini Research and Technology: 2B and 7B open decoders with pretraining, instruction and RLHF tuning, and benchmark results.
• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone: 3.8B model trained on filtered web and synthetic reasoning data, with SFT and DPO and an on-device build.
• Code Llama: Open Foundation Models for Code: Llama 2 continued on code, with infilling, long-context fine-tuning and instruction variants.
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling: documented 22-source English corpus used for pretraining and comparison.
• Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning: fine-tuning in a random low-dimensional subspace; better-pretrained models need fewer dimensions.
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing: survey that frames supervised NLP as prompting a pretrained model; no experiments.
• TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data: BERT-initialized encoder pretrained on text-table pairs, reading sampled rows with vertical attention; the code is non-commercial.
• Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus: documentation of C4 by source, contents and removed text; advises against blocklist filtering.
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey: survey of pre-train-then-fine-tune, prompt-based and text-generation uses of pretrained models; no experiments.
• Textbooks Are All You Need: phi-1, a 1.3B Python model trained on filtered web code, synthetic textbooks and exercises.
• Spanish Pre-trained BERT Model and Evaluation Data: BETO, a Spanish-only BERT-Base with whole-word masking, and the GLUES task suite.
• Large Language Models: A Survey: broad survey of how LLMs are built, used and benchmarked; compiled tables from third-party results.
• LinkBERT: Pretraining Language Models with Document Links: BERT-style encoder pretrained on linked document pairs with a three-way relation loss; biomedical variant on citation links.
• ProGen2: Exploring the Boundaries of Protein Language Models: decoder-only protein language models from 151M to 6.4B parameters, with zero-shot fitness ranking and sequence generation.
• CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X: 13B multilingual code model with the HumanEval-X benchmark and INT8 inference.
• BERT: A Review of Applications in Natural Language Processing and Understanding: review of BERT uses in NLP, from domain retraining to attacks; its tables quote the cited papers.
• Qwen Technical Report: 1.8B to 14B open decoders on up to 3T tokens, with ChatML SFT, a reward model and PPO, and code and maths variants; results are author-reported.
• ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic: two BERT-Base Arabic encoders, one on formal text and one on tweets, with the ARLUE benchmark.
• DeBERTaV3, a DeBERTa variant that uses ELECTRA-style pretraining with gradient-disentangled embedding sharing: replaced token detection with embedding sharing that stops gradients on the generator side.
• GLM-130B: An Open Bilingual Pre-trained Model: 130B English and Chinese blank-infilling model, with DeepNorm, FP16 stability tricks and INT4 weight quantization.
• Galactica: A Large Language Model for Science: decoder-only models up to 120B trained on a curated scientific corpus, with citation and working-memory tokens.
• StarCoder: may the source be with you!: 15.5B code model on permissively licensed code, with multi-query attention and fill-in-the-middle.
• MotionGPT: Human Motion as a Foreign Language: motion codes from a VQ-VAE added to a Flan-T5 vocabulary, with span-corruption pretraining and instruction tuning.
• Can Language Models Learn to Listen?: GPT-2 Medium predicts 256 VQ listener-motion codes from a speaker's timed transcript; text-only pretraining is ablated.
• A Comprehensive Overview of Large Language Models: survey of LLM building blocks, models, tuning, efficiency and evaluation; no experiments.
• ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools: GLM-4 technical report, with grouped-query attention, long context, alignment and a model that picks its own tools.
• COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter: BERT-Large continued on 22.5M COVID-19 tweets with masked-LM and next-sentence losses, then fine-tuned on five classification sets.
• LongT5: Efficient Text-To-Text Transformer for Long Sequences: T5 encoder with sliding-window and transient global attention, pretrained with principle-sentence generation, for inputs up to 16k tokens.
• MEDITRON-70B: Scaling Medical Pretraining for Large Language Models: Llama-2 continued on a medical corpus of about 47B training tokens with 1% general replay, then fine-tuned per benchmark, with three inference modes.
• KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media: language-specific BERT-base with a five-width CNN over its last four layers, for offensive-tweet detection in Arabic, Greek and Turkish; also releases ArabicBERT.
• Transformer Quality in Linear Time: FLASH, a gated single-head attention block with exact attention inside chunks and linear attention across them, for linear-cost language modeling.
• Spam and Sentiment Detection in Arabic Tweets Using MARBERT Model: MARBERT fine-tuned for five-class sentiment and for spam on private telecom tweets; the rare sarcasm class stays weak.
• BERTje: A Dutch BERT Model: BERT-base trained only on Dutch text, with sentence-order prediction and a 12 GB corpus; compared with multilingual BERT.
• MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction: continued masked-LM pretraining of a science model on materials full text, with NER and classification heads.
• A New Generation of Perspective API: Efficient Multilingual Character-level Transformers: token-free byte-level encoder for multilingual toxicity scoring, with a learned byte-block tokenizer.
• ChemBERTa-2: Towards Chemical Foundation Models: RoBERTa-style SMILES models pretrained with masked LM or descriptor regression and compared on MoleculeNet.