Masked language modeling

Overview

Masked language modeling (MLM) hides some units of an input and trains a model to recover the hidden units from the visible ones. The loss is taken only at the hidden positions 1911.03894 §4.3.

The cards here use the same idea on different inputs and for different goals:

• Text pretraining. An encoder is pretrained by recovering masked word pieces, here in French 1911.03894 §4.3 1912.05372 §3.2, or trained on German text only 2012.02110 §3.
• Arabic text. One encoder is pretrained on formal Arabic with whole-word masking, and another on Arabic tweets without the next-sentence objective 2101.01785 §3.1 2101.01785 §3.2.
• Dutch text. A BERT-base encoder is pretrained on Dutch text alone, with sentence order prediction in place of next-sentence prediction. The paper describes whole-word masking, but the released code switches it off 1912.09582 §2.2 create-pretrain-data.py.
• Molecules. A RoBERTa-style encoder reads SMILES strings. It is pretrained with MLM, or with regression on about 200 computed properties, which the authors compare directly 2209.01712 §3.1 2209.01712 tab. 1.
• Linked documents. An encoder is pretrained on two-segment inputs. The second segment is the next one from the same document, a random one, or one from a linked document. A three-way relation head is trained next to MLM 2203.15827 §4.2.
• Parallel text. Translation language modeling masks words in a pair of parallel sentences, and the model may use either sentence to fill a gap 2103.11943 p. 9.
• Clinical records. A Transformer over patient histories is pretrained by recovering masked diagnosis codes 1907.09538 §2.4.
• Tabular data. A Transformer over categorical columns is pretrained by recovering masked feature values 2012.06678 §2.
• Table and text. A BERT-initialised encoder reads a question with sampled table rows, and is pretrained with MLM on the text plus masked column and cell objectives 2005.08314 §3.2.
• Vision-language. A fusion encoder that reads caption tokens and image features is trained with MLM, optionally with a teacher's soft targets 2107.07651 §3.2 2107.07651 §3.3.
• Self-distillation. A student regresses teacher representations at masked positions, with text as one of three modalities 2202.03555 §3.3.
• Labels. A multi-label image classifier hides some of its label inputs and predicts them 2011.14027 §3.4.
• Querying. A fixed masked model is not trained further. Instead a cloze-style prompt with a blank is learned, so the model fills in the blank with a fact 2104.06599 §3.
• Probing. A masked encoder fills a blank in a fixed sentence, and the fill-in words are scored against a stance lexicon 2305.08283 §2.1.
• Generation. A masked model starts from an all-blank sequence and fills blanks over several rounds. The cards use this for discrete motion tokens 2312.00063 §3.4 and, as masked diffusion, for text 2406.07524 §4.1.
• Evaluation. Masked encoders are fine-tuned with a multiple-choice head and compared with decoder models on a reading benchmark 2308.16884 §4.2.
• Sentence embeddings. Words are deleted from a sentence, and a decoder rebuilds the sentence from one encoder vector. The paper compares this with MLM as an unsupervised pretraining step 2104.06979 §3.1 2104.06979 §5.2.
• Replaced-token detection. A small generator fills masked slots, and a larger discriminator labels every token as original or replaced. The generator is trained with MLM; the discriminator takes the place of the masked-token loss 2111.09543 §3.1.
• Blank infilling. A bidirectional model hides spans of its input, and writes the spans back. Most training sequences keep a prefix and hide a tail 2210.02414 §2.1 2210.02414 §2.2.
• Graph-text training. A text-graph encoder hides spans of its inputs as augmentation during a warm-up stage 2105.02605 §3.3. A separate encoder is pretrained on text and a knowledge graph with MLM and link prediction together 2210.09338 §2.3.
• Bias measurement. A masked model scores candidate entities in the blank of a culturally framed prompt 2305.14456 §4.4. A masked model also scores stereotype and anti-stereotype fill-ins in a sentence blank 2004.09456 §7.1.
• Scoring. A masked model is scored by pseudo-likelihood on grammatical minimal pairs 2504.08165 §5.2.1.
• Adversarial text. A masked model proposes replacement or inserted words that can flip a text classifier's label 2004.01970 §2.
• Vision-language with masked patches. A fusion encoder predicts masked caption words and masked image-patch codes, with image-text matching on the same pass. Unimodal MLM and masked image modeling run on unpaired data 2112.04482 §3.2 2112.04482 §3.3.
• Ad headlines. A BERT-Large model hides headline tokens during training, then is tuned with a reward on sampled headlines 2607.06818 §3.1 2607.06818 §3.3.
• Long context. A linear-cost attention model is pretrained on C4 with MLM as one of its settings 2202.10447 tab. 12.

The invariant across the MLM-style training variants is the scoring rule. The model is judged only on the hidden units, and the target is the original value 1911.03894 §4.3 2210.09338 §2.3. The masked diffusion and masked generation variants keep this rule: unmasked positions contribute nothing to the loss 2406.07524 §3.5 2312.00063 §3.2. The tabular, German and Spanish variants keep it too 2012.06678 §2 2012.02110 §3 2308.02976 §3. The table-and-text variant keeps MLM on the context and adds column and cell targets on top 2005.08314 §3.2. LinkBERT keeps the rule for its MLM term, and adds a relation loss on the [CLS] vector 2203.15827 eq. 1. Translation language modeling also keeps the rule, applied to a pair of parallel sentences 2103.11943 p. 9. ARBERT and MARBERT keep the rule for their masked-LM objective, with MARBERT dropping next-sentence prediction 2101.01785 §3.1 2101.01785 §3.2. The FLAVA masked multimodal loss keeps the rule, but the released code applies it only to image-text pairs that are matched flava.py. The GLM-130B loss is taken on the hidden spans 2210.02414 §2.1.

Six variants depart from the rule. The sentence-embedding variant deletes tokens and rebuilds the whole sentence 2104.06979 §3.1. The graph warm-up uses a contrastive link loss on masked inputs 2105.02605 §3.3. data2vec regresses averaged teacher block outputs at the masked steps, not the token 2202.03555 §3.3. The momentum variant mixes the one-hot token with a teacher distribution 2107.07651 §3.3 2107.07651 eq. 7. The replaced-token discriminator scores every token, not only the masked ones 2111.09543 §3.1. The self-critical stage of the ad-headline model replaces the token target with a reward-weighted sampled token at the hidden positions 2607.06818 eq. 9. The querying, probing, bias, scoring and adversarial uses take the model without training on the hidden units.

Variants

Subword masking

The baseline in the CamemBERT ablation. Individual subword tokens are the masking units 1911.03894 §4.3. In the authors' BASE model trained on OSCAR for 100k steps, it scores UPOS 97.78, LAS 89.80, NER 91.55 and XNLI 81.04 1911.03894 tab. 8. Card: CamemBERT: a Tasty French Language Model.

FlauBERT also masks subword units, with masks redrawn dynamically. It pretrains with MLM only, on a 71 GB French corpus 1912.05372 tab. 1 1912.05372 §3.1. The masking rate is not stated in the text 1912.05372 tab. 1. Card: FlauBERT: Unsupervised Language Model Pre-training for French.

Whole-word masking

Sample about 15% of the whitespace-delimited words, and hide all subword pieces of each sampled word. The authors also redraw the masks every time a sequence is used (dynamic masking), and drop next-sentence prediction 1911.03894 §4.3. Card: CamemBERT: a Tasty French Language Model.

In the same ablation, whole-word masking scores UPOS 97.79 and LAS 89.88 against 97.78 and 89.80 for subword masking. It scores XNLI 81.55 against 81.04, but NER 91.44 against 91.55 1911.03894 tab. 8. The authors summarise the effect as +0.5 on XNLI and little elsewhere 1911.03894 app. A.

Whole-word masking needs the mapping from subword pieces back to words, which the paper defines on whitespace in the untokenised text 1911.03894 §4.3.

Dynamic masking in a RoBERTa-style recipe

Dynamic masking draws a new mask each time a sequence is used. The original BERT used static masking 2103.11943 p. 11. The survey lists dynamic masking as one of RoBERTa's changes, with more text (about 160 GB from four corpora), no next-sentence prediction and larger batches [arxiv:2103.11943 p. 11–12]. Those changes together are reported as a gain of +3 to +4 points over BERT. The survey does not isolate the masking change, and it does not state the masking rate 2103.11943 p. 11–12. The RoBERTa paper has no card here, so these figures are the survey's account of it.

Monolingual German encoder

GottBERT is a RoBERTa-base or RoBERTa-large encoder trained only on German text, with fairseq 2012.02110 §3. Its pretraining is a RoBERTa-style masked LM 2012.02110 §3. The paper gives no masking rate and does not say whether masking is per subword or per word. Whole-word masking appears only as future work 2012.02110 §5 2012.02110 §6. Pretraining runs 100k updates with batch size 8k and a peak learning rate of 4e-4 for base 2012.02110 §3. On CoNLL 03 the best filtered base model scores 86.14 F1, against 85.37 for GELECTRA base 2012.02110 tab. 3. The paper reports single runs, so differences of about one point are within unknown noise 2012.02110 §5. Card: GottBERT: a pure German Language Model.

Monolingual Spanish encoder with whole-word and dynamic masking

BETO is a BERT-Base-shaped encoder with 12 layers, hidden size 768 and 110M parameters, pretrained with MLM on Spanish text only 2308.02976 §3. It trains a cased and an uncased model 2308.02976 §3. Masking is whole-word: when one sub-word of a word is hidden, its contiguous sub-words are hidden too. It is also dynamic, with 10 different masks per sentence 2308.02976 §3. The masking rate and the replacement rule are not stated 2308.02976 §3. The corpus is about 3 billion words from Spanish Wikipedia and OPUS sources, including subtitles and news 2308.02976 §3. Pretraining runs 2M steps. The first 900k use batch size 2048 and length 128, and the rest use batch size 256 and length 512. The peak learning rate is 1e-4, after 10,000 warm-up steps 2308.02976 §3. The authors also compile Spanish tasks into GLUES 2308.02976 §4. Author-reported, BETO-cased scores 82.01 on XNLI and 88.43 on NER, against 78.50 and 87.38 for the best multilingual BERT the authors cite. Uncased is ahead on PAWS-X (89.55 against 89.00) and MLDoc (96.12 against 95.70) 2308.02976 tab. 1. On one TAR-to-XQuAD setting, the best mBERT leads on F1, 77.60 against 77.56 2308.02976 tab. 2. The text reports 90.7 for mBERT trained on English PAWS-X data, a different setting from the 89.00 in Table 1 2308.02976 §5.2 2308.02976 tab. 1. The paper does not say whether its scores are dev or test scores 2308.02976 §5.2. The corpus draws on OPUS sources with mixed licences, so check those terms before commercial reuse of the weights or the corpus 2308.02976 §3. Card: Spanish Pre-trained BERT Model and Evaluation Data.

Monolingual Arabic encoders: formal text and tweets

ARBERT and MARBERT are BERT-Base-shaped encoders with the same architecture, differing in data and objective 2101.01785 §3.1 2101.01785 §3.2. ARBERT is pretrained on about 61 GB of mostly formal Arabic, with whole-word masking at 15%, 80/10/10 2101.01785 §3.1. MARBERT is pretrained on 1B sampled tweets, with next-sentence prediction dropped and its masking details not restated 2101.01785 §3.2. Both use a 100K WordPiece vocabulary 2101.01785 §3.1 2101.01785 §3.2. Author-reported, on the test splits of the ARLUE benchmark, the strongest entry is MARBERT-v2, at 77.40 against 76.55 for XLM-R Large 2101.01785 §5.2. The paper never defines MARBERT-v2 beyond the label, so treat it as a name only 2101.01785 §7. The MARBERT tweet data is in-house and will not be released, and the checkpoints are for research use only 2101.01785 §3.2. Card: ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic.

Monolingual Dutch encoder with sentence order prediction

BERTje is a BERT-base model trained on about 12 GB (about 2.4B tokens) of Dutch books, news, Wikipedia and SoNaR text 1912.09582 §2.1. Each training pair is either in order or swapped, and the swapped case is the sentence-order task that replaces next-sentence prediction [arxiv:1912.09582 §2.2]. Its masked-LM rule is 15% of tokens with 80/10/10. The paper says the hidden tokens are consecutive wordpieces of the same word, so whole words are masked together 1912.09582 §2.2. The released code turns whole-word masking off, and caps predictions at 20 per sequence, so a full-length text gets a masked share of about 4%, not 15% [code: pretraining/create-pretrain-data.py#L32] [code: pretraining/create-pretrain-data.py#L34]. The released run command sets 1M steps, batch size 256, peak learning rate 1e-4 and 10,000 warm-up steps run-pretraining.sh. The paper gives none of these.

Author-reported, on the CoNLL-2002 test split, BERTje scores 88.3 F1 against 80.7 for multilingual BERT 1912.09582 tab. 1. On UD-LassySmall test accuracy it scores 96.3 against 92.5 1912.09582 tab. 2, and on the 110k Dutch book-review test set 93.0 against 89.1 1912.09582 tab. 4. The paper reports single runs, so gaps of about a point are not established 1912.09582 §4. The README reports 90.24 on the CoNLL-2002 test set, which differs from the paper's 88.3 README.md. Card: BERTje: A Dutch BERT Model.

Translation language modeling across parallel sentences

Translation language modeling (TLM) is the cross-lingual objective from XLM (arXiv:1901.07291). It masks words in a pair of parallel sentences and lets the model use either sentence to fill a gap 2103.11943 p. 9. Without parallel text, the fallback is masked language modeling alone [arxiv:2103.11943 p. 10]. The survey reports average machine-translation gains over the best known models of +4.9% for TLM and +1.3% for MLM alone 2103.11943 p. 10. These figures are the survey's account of work it cites. The XLM paper has no card here, so check it on arXiv before quoting the numbers.

Masked diagnosis codes in patient records

Here the units are diagnosis codes in a patient's visit history. The method follows BERT: 15% of the tokens are selected, and of those, 80% become MASK, 10% a random vocabulary token, and 10% stay unchanged. The loss is computed only at the selected positions 1907.09538 §2.4. This pretraining is followed by a fine-tuning stage that predicts diagnoses at the next visit or within 6 and 12 months 1907.09538 §2.5. The paper reports pretraining precision at threshold 0.5, 0.6597 for the tuned architecture 1907.09538 §2.4 1907.09538 tab. 2. Its downstream comparison against DeepR and RETAIN is on UK primary-care data, and the authors report average precision and AUROC on that comparison 1907.09538 tab. 1. Card: BEHRT: Transformer for Electronic Health Records.

Masked feature prediction in tabular data

Here the units are the categorical features of one row. The method picks k% of them, here k = 30, and sets each to a missing-value embedding. A multi-class classifier then predicts the original class of each masked feature from its contextual embedding 2012.06678 §2. The paper does not say whether that classifier is shared across columns 2012.06678 §2. This pretraining is used only in semi-supervised runs 2012.06678 §3. On the 8 datasets with more than 30K rows, the MLM variant scores 66.8, 71.0 and 72.9 mean AUC at 50, 200 and 500 labels, against 66.6, 70.9 and 73.1 for the replaced-value variant 2012.06678 tab. 3. On the 12 smaller datasets, MLM scores 78.5, 81.0 and 82.4, against 78.6, 81.6 and 83.4 for the replaced-value variant. The best other method at each label count scores 79.4, 81.2 and 82.3, and the paper names a different method at each count 2012.06678 tab. 4. Card: TabTransformer: Tabular Data Modeling Using Contextual Embeddings.

Masked context and column objectives for table-and-text pretraining

TaBERT encodes an utterance with K rows of a table. Each row is one BERT input: the utterance tokens, then one string per cell holding the column name, a type and the cell value 2005.08314 §3.1. It starts from uncased BERT-Base or BERT-Large 2005.08314 §5. Pretraining sums three losses [arxiv:2005.08314 app. A.2]. The first is MLM on the context, at 15% of sub-tokens 2005.08314 §3.2. The second is masked column prediction: 20% of columns have their names and types hidden in every row, and the model predicts the hidden tokens from the column vector 2005.08314 §3.2. The third is cell value recovery, which predicts the value tokens of the hidden columns from each cell's vector with a span head. Cell values stay visible in the input 2005.08314 §3.2. The paper does not state the replacement rule for context masking 2005.08314 §3.2. Training data is 26.6M examples, 1.3M from Wikipedia tables and 25.3M from the WDC WebTable Corpus, each with nearby text 2005.08314 §3.2 2005.08314 app. A.1.

The objective ablation on TaBERT-Base with K = 3 compares MCP alone with MCP plus cell value recovery. On WikiTableQuestions dev the scores are 51.6 ± 0.7 and 51.6 ± 0.5. On Spider dev they are 62.6 ± 0.7 and 63.3 ± 0.7. The authors call the cell-value gain slight 2005.08314 tab. 5 2005.08314 §5. Author-reported, MAPO with TaBERT-Large and K = 3 reaches 52.3% test accuracy for the run that was best on dev, against 46.9% for the earlier ensemble of 10 2005.08314 tab. 1. The mean over runs is 51.8 ± 0.6 [arxiv:2005.08314 tab. 1]. The authors say some WikiTableQuestions test tables may be in the pretraining corpus, but judge the share tiny 2005.08314 §5. The largest model took about six days on 128 V100 GPUs 2005.08314 app. A.2. The released code is non-commercial, so this section relies on the paper alone. Card: TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data.

Label masking for multi-label classification

Here the units are labels rather than tokens. For each training image, the method picks how many labels to hide: a number between 0.25 and 1 times the number of labels 2011.14027 §3.4. Hidden labels get an "unknown" state in the input, and the other labels keep their ground-truth state. The loss is binary cross-entropy on the hidden labels only 2011.14027 §3.4.

The paper justifies the lower bound by comparison with typical MLM masking rates, and by the assumption that inputs with more than 75% of labels known are unrealistic 2011.14027 §3.4. Because the model learns to accept known states, labels known at test time can be fed in for a single forward pass 2011.14027 §4.2. The authors report 85.1 mAP on COCO-80 2011.14027 tab. 1. Card: General Multi-label Image Classification with Transformers.

Before reproducing this, read the card's Paper vs code section. The released loader switches label masking off for two of the datasets [code: dataloaders/data_utils.py#L17-L23].

Masked span warm-up for textual graphs

GraphFormers nests a graph-attention step in each layer of a BERT-style encoder, so a node's text is encoded together with its neighbours' text. Its training has two stages 2105.02605 §3.3. The warm-up stage runs on inputs where 15% of tokens are masked in spans, with the 80/10/10 rule 2105.02605 app. A.1. The loss in that stage is the contrastive link-prediction loss of 2105.02605 eq. 7, not recovery of the masked tokens. The masks therefore act as data augmentation. The fine-tuning stage then runs on unmasked data. Without the progressive stage, the paper reports P@1 drops of 0.98, 1.71 and 1.18 points on Product, DBLP and Wiki 2105.02605 §4.4 2105.02605 tab. 4. Card: GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph.

Joint masked and graph link prediction

DRAGON pretrains one encoder on a text segment and a local knowledge-graph subgraph. Two losses are summed with weight 1 each. The first is MLM on the text, with cross-entropy at the masked positions. The second is link prediction on held-out graph edges 2210.09338 §2.3 2210.09338 eq. 8. The pretraining masks 15% of text tokens and holds out 15% of edges 2210.09338 §3.1. In the authors' ablation, MLM alone scores 74.3 on CSQA and 67.2 on OBQA. Link prediction alone scores 73.8 and 66.4. The joint objective scores 76.0 and 72.0 2210.09338 tab. 5. The general model took 7 days on eight A100 GPUs in FP16 2210.09338 §3.1. The "RoBERTa" baseline in the paper is RoBERTa-Large continued with MLM on the same text and steps, not the released checkpoint 2210.09338 §3.3. Card: Deep Bidirectional Language-Knowledge Graph Pretraining.

Two-segment inputs with document links

LinkBERT builds a graph of documents from hyperlinks, or from citations in biomedical papers 2203.15827 §4.1. Each training input holds an anchor segment and a second segment. The second segment is the next one of the same document, a random one, or one from a document the anchor links to. The three options are equally likely in the main setting 2203.15827 §4.2 2203.15827 §5.1. A linked document is drawn with probability inversely proportional to its in-degree 2203.15827 §4.3. The model keeps MLM at 15% with the 80/10/10 rule 2203.15827 §3. It adds a three-way relation head on the [CLS] vector, which replaces next-sentence prediction, and the two losses have weight 1:1 2203.15827 §4.2 2203.15827 eq. 1.

Author-reported, MRQA average F1 is 77.8 for LinkBERT-base against 75.2 for BERT-base, and 81.0 against 78.5 for large 2203.15827 tab. 1. Without the relation loss, base scores fall on the datasets reported, for example HotpotQA from 78.2 to 76.5 2203.15827 tab. 6. The authors state that a random-link tiny model matches BERT-tiny 2203.15827 §5.5. Base models took four days on four A100 GPUs, and the biomedical base took seven days on eight 2203.15827 §5.1 2203.15827 §6.1. The cost of building the graph is not stated 2203.15827 §6.1. The released repository holds fine-tuning and preprocessing scripts, not the pretraining loop, so the link sampling and relation head cannot be checked against code. Card: LinkBERT: Pretraining Language Models with Document Links.

Masked fusion with momentum distillation

ALBEF first aligns image and text features with a contrastive loss 2107.07651 §3.2, then runs a fusion encoder in which the text tokens attend to image tokens 2107.07651 §3.1. Its MLM loss selects 15% of caption tokens, with the 80/10/10 rule 2107.07651 §3.2. The target is the one-hot vocabulary label 2107.07651 eq. 3. The momentum variant mixes this loss with a KL term to an EMA teacher's distribution over the vocabulary, at each masked position, with weight α = 0.4 2107.07651 eq. 7 2107.07651 §3.5. On 4M pretraining images, the momentum term on MLM raises VQA test-dev from 74.06 to 74.42 and NLVR² test-P from 80.34 to 80.44 2107.07651 tab. 1. The card notes that the EMA copy doubles weight memory, because the teacher's encoders run in a second forward pass each step model_pretrain.py. Card: Align before Fuse: Vision and Language Representation Learning with Momentum Distillation.

Sentence reconstruction by token deletion

TSDAE is not masking in the BERT sense. It deletes words from a sentence and trains a decoder to rebuild the sentence from the encoder's single vector. The decoder's cross-attention sees only that vector 2104.06979 §3.2. The best deletion ratio in the appendix is 0.6, meaning 60% of tokens are affected 2104.06979 app. A 2104.06979 tab. 6. After training the decoder is discarded, and the encoder alone produces embeddings 2104.06979 §3.2. Over four domain tasks, the unsupervised average is 55.2 for TSDAE and 52.9 for MLM with BERT-base 2104.06979 tab. 2. In the appendix tuning run, deletion at ratio 0.3 scores 78.33 on STS-B and masking scores 76.56 2104.06979 app. A 2104.06979 tab. 5. Card: TSDAE.

Replaced-token detection with gradient-disentangled embedding sharing

DeBERTaV3 swaps DeBERTa's masked-token objective for ELECTRA-style replaced token detection 2111.09543 §3.1. A generator, half the discriminator's depth, is trained with MLM on about 15% of tokens. Its sampled replacements are fed to a discriminator that labels every token as original or replaced 2111.09543 eq. 3 2111.09543 eq. 4. The paper's own contribution is the embedding sharing rule. The discriminator's token embedding is the generator's with its gradient stopped, plus a residual that only the discriminator trains 2111.09543 §3.3.

In a 125k-step ablation, vanilla sharing scores 88.8/88.4 on MNLI-m/mm, no sharing 88.3/87.9, and the gradient-disentangled version 89.3/89.0 2111.09543 tab. 2. Author-reported, DeBERTaV3-Large reaches a GLUE dev average of 91.37, against 90.00 for DeBERTa-large and 89.46 for ELECTRA-large 2111.09543 tab. 3. The ablations use a smaller pretraining regime than the full runs, so the two sets of numbers are not comparable 2111.09543 §4. The paper weights the discriminator loss by λ = 50 2111.09543 §3.1, while the released code defaults to 10 rtd_task.py. Card: DeBERTaV3: ELECTRA-style pretraining with gradient-disentangled embedding sharing.

Masked multimodal and unimodal objectives on public pairs

FLAVA has an image encoder, a text encoder and a six-layer multimodal encoder 2112.04482 §3.1. On image-text pairs, its masked multimodal loss masks rectangular blocks of image patches, tokenised by a pretrained dVAE, and 15% of text tokens. The multimodal encoder predicts both sets of masked targets 2112.04482 §3.2. On unpaired text it runs MLM at 15%, and on unpaired images masked image modeling on dVAE tokens 2112.04482 §3.3. The text encoder starts from MLM on CCNews and BookCorpus, and the image encoder from DINO 2112.04482 §3.3 2112.04482 app. C.

Author-reported, the full model reaches a macro average of 75.85 over 35 tasks, against 66.78 for CLIP ViT-B/16 trained on 400M pairs 2112.04482 tab. 3 2112.04482 tab. 4. Keeping only the multimodal losses gives 74.23, and removing the unimodal initialisation gives 73.72 2112.04482 tab. 3. The column order in table 3 is wrong for some rows, so take per-modality numbers from table 4 2112.04482 tab. 4. On ImageNet linear evaluation the released CLIP model is ahead, 80.2 against 75.5 2112.04482 tab. 5, so the masked objectives do not win on vision alone. Card: FLAVA: A Foundational Language And Vision Alignment Model.

Masked regression to teacher features

data2vec uses one recipe for speech, images and text. An EMA teacher sees the unmasked input and averages the outputs of its top K = 10 blocks, with each block normalised 2202.03555 §4.3 2202.03555 §3.3. A student sees the masked input and regresses those averages at the masked steps with a smooth L1 loss, beta 4 2202.03555 §3.3 2202.03555 §4.3. The target is the teacher's representation, not the token, so no tokenizer is needed for the target 2202.03555 §3.3. The paper's main text setup is BERT masking, 15% with the 80/10/10 rule 2202.03555 §4.3. The released text config uses spans of 4 at probability 0.35, with no random or unmasked tokens base.yaml. On GLUE dev, the single-task average is 82.7 for data2vec and 82.5 for a RoBERTa baseline retrained in the same setup 2202.03555 tab. 4. Card: data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language.

Cloze prompts for a fixed masked model

This is an inference-side use. The masked model is frozen. A relation such as date of death is queried with a cloze prompt that has a blank for the subject and a blank for the object 2104.06599 §3. The paper replaces the prompt tokens with continuous vectors, adds a trainable perturbation at every layer, and fits a mixture of such prompts with EM for each relation [arxiv:2104.06599 §3.1–§3.5]. On T-REx with BERT-large, the mixture reaches P@1 51.6, against 39.4 for the LPAQA baseline, which the authors reimplemented [arxiv:2104.06599 tab. 1]. Card: Learning How to Ask: Querying LMs with Mixtures of Soft Prompts.

Masked diffusion over text

Masking-only diffusion treats masking as the noise process. Each token is replaced by the mask token independently, with a probability set by a schedule, and a masked token stays masked 2406.07524 §3.1 2406.07524 §3.2.1. The denoiser is a diffusion transformer with timestep conditioning 2406.07524 §3.5.1, and its training input is the masked sequence, so the step is close to MLM. The paper's continuous-time loss is a weighted average of masked-token losses 2406.07524 §3.5. In the experiments the masking probability at time t is t, which gives a weight of 1/t 2406.07524 app. D.2. That weight is derived from the paper's eq. 10 and is not written out there 2406.07524 §3.5.

Generation starts from a row of mask tokens and unmasks over N steps. Each masked token is revealed with a probability set by the schedule, and the unmasking is random rather than ranked by confidence 2406.07524 §4.1. The code's default is 128 sampling steps, and its OWT generative-perplexity script uses 1,000 config.yaml [code: scripts/train_owt_mdlm.sh#L27]. Longer text uses a block-by-block variant with a clean prefix 2406.07524 §4.2. Card: Simple and Effective Masked Diffusion Language Models.

The reported likelihoods are upper bounds. On LM1B the 327B-token bound is 23.00, against 20.86 for a retrained autoregressive model [arxiv:2406.07524 tab. 1]. On OpenWebText the bound is 23.21, against 17.54 for autoregressive 2406.07524 tab. 2. Bounds from diffusion models are not directly comparable with exact autoregressive likelihoods 2406.07524 tab. 1. The semi-autoregressive sampler reaches a generative perplexity of 27.18 at 89.3 seconds per sequence, against 35.43 at 2473.9 seconds for SSD-LM, on 200 sequences of 2,048 tokens on one GPU 2406.07524 tab. 5.

Iterative masked decoding over discrete tokens

MoMask uses masking for the base layer of a residual quantizer. A bidirectional transformer over base tokens is conditioned on CLIP text features. The mask ratio follows a cosine schedule, and the loss is the negative log-likelihood of the original tokens at masked positions 2312.00063 §3.2 2312.00063 eq. 3. At inference, all base tokens start masked. The model fills them in parallel over L iterations, and the least confident fills are masked again on each round. The paper uses L = 10 2312.00063 §3.4 2312.00063 §4. A second bidirectional transformer then predicts the finer residual layers one at a time 2312.00063 §3.4. The introduction says all layers are produced in about 15 iterations 2312.00063 §1.

On HumanML3D, the paper reports FID 0.045 for MoMask against 0.141 for T2M-GPT 2312.00063 tab. 1. The ablation table lists 0.051 for the same setting 2312.00063 tab. 2. The authors report that generation quality starts to degrade above five residual layers 2312.00063 §4.2. The same inference procedure, with a time range masked, fills in a gap inside an existing motion 2312.00063 §4.3. Card: MoMask: Generative Masked Modeling of 3D Human Motions.

Self-critical tuning of a masked headline generator

The ad-headline model is a BERT-Large model over a set of product titles. Its hidden positions are only in the headline, and the source titles are never masked in the main model 2607.06818 §3.2. Stage 1 trains with the masked-token cross-entropy over those positions 2607.06818 §3.1 2607.06818 eq. 2. The masking rate is not stated 2607.06818 §3.1. Stage 2 is self-critical. For each campaign, one headline is sampled and one is decoded greedily. The sampled headline is pushed up only when its ROUGE-L F1 against the approved headline beats the greedy one 2607.06818 §3.3 2607.06818 eq. 9 2607.06818 §4.1. Inference uses beam search with a length-normalised score 2607.06818 §3.4 2607.06818 eq. 10.

The paper reports ROUGE-L 6.33 above a bi-LSTM baseline for the full model, and 5.08 for the MLM stage alone 2607.06818 tab. 1. Its table values are differences from that baseline. The reward is the same metric as the top overlap row, so part of that gain reflects the metric being optimised. The crowd audit is the independent check: 6.53% more headlines are rated 3 of 3 than human-submitted ones 2607.06818 tab. 2. The data is proprietary, English only 2607.06818 §4.1, and the card records no official code. Card: Ad Headline Generation using Self-Critical Masked Language Model.

Mask-filling probe of an encoder

To ask whether a masked encoder leans politically, the probe places each political statement inside a fixed request with one mask for the stance word. The model returns its ten most likely fill-ins. The mass on a positive lexicon is compared with the mass on a negative one, and the answer is "strong" if the gap is at least 0.3 2305.08283 §2.1 2305.08283 app. A.1. The lexicons were built by hand from 100 example outputs, and the 0.3 threshold was set by hand 2305.08283 app. A.1. The repository has no encoder code for this probe step1_response.py.

The same paper then continues pretraining RoBERTa on partisan corpora and fine-tunes it on hate-speech and misinformation tasks 2305.08283 §4.1 2305.08283 §4.2. Overall gaps are small. On Hate-Demographic, balanced accuracy is 90.26 for vanilla RoBERTa and 89.30 for the right-leaning news variant 2305.08283 tab. 3. The per-group differences are the paper's main result 2305.08283 §4.2 2305.08283 tab. 4. Card: From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models.

Masked-entity bias benchmark

CAMeL replaces a culture-specific entity with a mask token in each of its prompts 2305.14456 §3.2 2305.14456 tab. 1. It has 628 Arabic prompts, 250 of them contextualised to Arab culture 2305.14456 §3. For BERT-type models the full prompt is scored with its mask token. For GPT- and T5-type models only the text before the mask is used 2305.14456 §4.4. The Cultural Bias Score is the share of Arab–Western entity pairs where the model rates the Western entity higher, and zero is ideal 2305.14456 §4.4. Author-reported, on the contextualised prompts, ARBERT scores 46.40 and AraBERT-T (base) scores 50.27 2305.14456 tab. 10. The paper rewrote some prompts for GPT-type models so the Arab cue comes before the mask, so their scores use different prompts from the masked-LM scores 2305.14456 app. C.1. The prompts derive from Twitter/X posts, and the paper says they are masked before release, so check the data's terms before reuse 2305.14456 Ethics Statement. Card: Having Beer after Prayer? Measuring Cultural Bias in Large Language Models.

Stereotype fill-in scoring

StereoSet's intrasentence test has a blank in a context. Each of its three options is a stereotype, an anti-stereotype or an unrelated option 2004.09456 §2 2004.09456 §2.1. For BERT, RoBERTa and XLNet, each attribute subword is scored with its own masked prompt, with earlier subwords visible dataloader.py. The card's paper-vs-code notes say the paper averages log probabilities, while the code averages raw probabilities 2004.09456 §7.1 eval_discriminative_models.py. Both sub-scores only rank options within each item 2004.09456 §6.2. ICAT combines a language-model score and a stereotype score 2004.09456 §6.2. The author-reported dev-split ICAT is 69.4 for BERT-base and 71.9 for GPT-2 small 2004.09456 tab. 4. The test split is hidden 2004.09456 §6.1, and the repository has no test CATs . Card: StereoSet: Measuring stereotypical bias in pretrained language models.

Multilingual evaluation with masked encoders

Belebele is a test set, not a training method. Its 900 four-option reading items are parallel across 122 language variants 2308.16884 §3.5. Masked encoders are evaluated in two ways: fine-tuned on English, or fine-tuned on machine-translated training data for every target language 2308.16884 §4.2 2308.16884 app. A.4.1. For XLM-V, the translate-train setting averages 60.2 across languages, against 55.6 for English-only fine-tuning 2308.16884 tab. 2. LLaMA 2 70B with five English examples averages 48.0 over the same languages, although it reaches 90.9 on English 2308.16884 tab. 2.

The benchmark is CC BY-SA, but its English training set is assembled from CC BY-NC sources, so commercial fine-tuning on it is not permitted 2308.16884 app. A.3. Use the test set only, and do not reuse the training set for commercial models 2308.16884 §3.4. Card: The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants.

Pseudo-likelihood scoring of masked models

The BabyLM shared task scores grammar with zero-shot minimal pairs. The model is correct when it gives the acceptable sentence the higher score 2504.08165 §5.1. Masked models are scored by pseudo-likelihood, following Salazar et al. (2020), through the minicons implementation 2504.08165 §5.2.1. The paper's organisers scored 31 teams' submissions on the same pipeline, and the reported scores are author-reported 2504.08165 §5.3. The card does not state whether the baseline models were trained with MLM, so this variant is evaluation only 2504.08165 §5.4. Card: Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora.

Masked-LM fillers for black-box adversarial text

BAE attacks a text classifier in the black-box, soft-label setting. The attacker may query output probabilities, but not parameters, gradients or training data 2004.01970 §2. Each token is scored by how much deleting it lowers the probability of the true label, following TextFooler and Ren et al. 2004.01970 §2. Tokens are visited in that order. Each is masked, and the top 50 predictions of a pretrained BERT base-uncased masked LM become candidates 2004.01970 §2 2004.01970 app. A. Candidates pass a Universal Sentence Encoder similarity filter with a threshold of 0.8, and replacements also pass a part-of-speech check 2004.01970 app. A. The search stops when the label flips 2004.01970 alg. 1. The four modes are BAE-R (replace), BAE-I (insert left or right of a token), BAE-R/I (choose one per token) and BAE-R+I (replace, then insert at the same position) 2004.01970 §2. The paper does not state how the insert side is chosen, or how BAE-R/I chooses between replace and insert 2004.01970 §2.

Author-reported results cover seven datasets and three targets: word-LSTM, word-CNN and BERT base-uncased 2004.01970 §3 2004.01970 tab. 1 2004.01970 tab. 2. BAE-R+I gives the lowest accuracy in most settings. BAE-I has the highest average similarity among the four modes 2004.01970 §3. For BERT as the target, TextFooler gives lower accuracy than BAE-R on six of seven datasets. BAE-R is the stronger attack only on TREC 2004.01970 tab. 1 2004.01970 tab. 2. In a human study on three datasets with the BERT target, raters scored BAE-R examples higher on naturalness than TextFooler examples, 3.91 against 3.17 on Amazon, on a 1 to 5 scale 2004.01970 tab. 3 2004.01970 app. C.

The early-stopping step uses the test split, so reported accuracy may be optimistic 2004.01970 app. A. Amazon, Yelp and IMDB each have 100 test points, so one point moves accuracy by about one percentage point 2004.01970 tab. 5. Card: BAE: BERT-based Adversarial Examples for Text Classification.

Blank infilling at 130B scale

GLM-130B is a bidirectional model trained by blank infilling 2210.02414 §2.1. In 30% of training sequences, several short spans are marked with [MASK]. Span lengths are drawn from Poisson(3), and about 15% of tokens are hidden 2210.02414 §2.2. In the other 70%, a prefix is kept, and the tail is hidden behind one [gMASK] span with a uniformly drawn length 2210.02414 §2.2. The spans are written back in a random order, and each one sees the spans already written 2210.02414 §2.1.

The paper's training-stability fixes are DeepNorm Post-LN, an embedding gradient shrink of 0.1, and FP16 training 2210.02414 §3. Author-reported, zero-shot LAMBADA accuracy is 80.21, against 76.2 for GPT-3 175B 2210.02414 §5.1. Weight-only INT4 quantization gives 79.47 on LAMBADA and 44.80 on MMLU, against 80.21 and 44.75 in FP16 2210.02414 §4. The released weights carry a non-commercial model license, separate from the Apache-2.0 code, so check the card's Pitfalls before reuse. Card: GLM-130B: An Open Bilingual Pre-trained Model.

Masked-LM pretraining with linear-cost attention

FLASH replaces the Transformer block with a gated attention unit, and splits attention into exact attention inside fixed-size chunks and linear attention between chunks 2202.10447 §2 2202.10447 §3.2. Its C4 MLM setting is one of the training regimes in the paper 2202.10447 tab. 6 2202.10447 tab. 7. At context 8192, the reported per-step latency on 64 TPU-v4 cores is 59.9 ms for FLASH and 252.1 ms for Transformer++, with C4 MLM perplexity 3.897 against 3.933 2202.10447 tab. 12. The training-cost speed-up over Transformer++ is 1.0x to 4.8x across contexts 512 to 8192 2202.10447 §4.1. These are wall-clock figures, not per-step speed-ups, and the card does not give the masking rate or loss for the C4 MLM runs. The card records no official code, so those details cannot be checked against one 2202.10447 tab. 6. Card: Transformer Quality in Linear Time.

Masked SMILES pretraining against descriptor regression

ChemBERTa-2 is a RoBERTa encoder over canonical SMILES strings, with a 591-token vocabulary and inputs of up to 512 tokens 2209.01712 §3.1. Its MLM step masks 15% of the tokens and predicts the originals [arxiv:2209.01712 §3.1]. The rival objective, multi-task regression (MTR), needs no masking: it predicts 200 RDKit properties computed from each string 2209.01712 §3.1. Both are run on 5M, 10M and 77M compounds 2209.01712 §3.

The authors' Table 1 compares the two at 77M. MTR-77M is ahead of MLM-77M on all eight columns, for example BACE RMSE 1.363 against 1.509 and Lipophilicity RMSE 0.798 against 0.987 2209.01712 tab. 1. The MLM rows also have ClinTox ROC-AUC values of 0.239 to 0.349, below chance, which the paper does not discuss 2209.01712 tab. 1. The authors conclude that MTR tends to fine-tune better 2209.01712 §4.1. The paper does not state the MLM replacement rule; the training code takes it from the HuggingFace collator utils.py. The paper gives no GPU-hour total 2209.01712 §7.1. Card: ChemBERTa-2: Towards Chemical Foundation Models.

Weak links

Three cards use masking or masked models without an MLM training objective. They are listed here so that a reader does not take them as MLM evidence.

• Full-sequence diffusion as partial masking. Diffusion Forcing reads full-sequence diffusion as partial masking of every token, and gives each token its own mask level 2407.01392 §3.1. It trains a causal recurrent denoiser, not a masked encoder, so it is a framing only. Card: Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion.
• Pretrained chemical language models. BOOM evaluates ChemBERTa, MoLFormer and Regression Transformer on out-of-distribution molecules. Pretraining improved in-distribution RMSE by 31%, 35% and 12%, with no significant out-of-distribution gain 2505.01912 §4.2. The card does not state these models' pretraining objectives. Card: BOOM.
• Fine-tuned MARBERT on Arabic tweets. MARBERT is the masked encoder here, but the method is a classifier fine-tuned from it. No masked-token step is described, so the card is evidence of downstream use, not of MLM training 2606.25495 §4.2 2606.25495 §5.2. The labelled spam set uses labels from a keyword list, so its high score may partly reflect that list 2606.25495 §5.1. Card: Spam and Sentiment Detection in Arabic Tweets Using MARBERT Model.

When to use which

The CamemBERT ablation runs use one pretraining seed per model, and the authors attribute some of the differences to seed noise 1911.03894 §6.2. Treat differences of a few tenths of a point as unresolved.

Implementation sketch

Five shapes appear in the cards. The first covers token and word masking 1911.03894 §4.3. Masked diagnosis codes follow the same rule, with a code in place of a token 1907.09538 §2.4.

# Select units to hide, one sequence  [arxiv:1911.03894 §4.3]
# units: groups of positions. One position each for subword masking;
# all subword pieces of one word for whole-word masking
def mask_sequence(tokens, units):
    targets = {}
    for unit in units:
        # Select about 15% of units  [arxiv:1911.03894 §4.3]
        if random_value() >= 0.15:
            continue
        # The loss is taken at every position of a selected unit  [arxiv:1911.03894 §4.3]
        for index in unit:
            targets[index] = tokens[index]
        # Draw once per unit: this is a choice of ours, see the notes below
        draw = random_value()
        for index in unit:
            if draw < 0.8:
                tokens[index] = MASK_TOKEN              # 80%  [arxiv:1911.03894 §4.3]
            elif draw < 0.9:
                tokens[index] = random_token()          # 10%  [arxiv:1911.03894 §4.3]
            # else: keep the original token              # 10%  [arxiv:1911.03894 §4.3]
    return tokens, targets

The second covers label masking in a multi-label classifier 2011.14027 §3.4.

# Hide a random share of the L labels for one training image  [arxiv:2011.14027 §3.4]
def label_mask_step(image, labels):    # labels: (L,) with 0 negative, 1 positive
    # Draw the number of hidden labels between 0.25L and L  [arxiv:2011.14027 §3.4]
    n_hidden = draw_count(low=0.25 * L, high=L)
    hidden = random_subset(L, n_hidden)
    # Hidden labels get the unknown state; the rest keep their ground truth  [arxiv:2011.14027 §3.4]
    state = labels.copy()
    state[hidden] = UNKNOWN
    logits = forward(image, state)     # one pass over all label tokens  [arxiv:2011.14027 §3.2]
    # Binary cross-entropy only on the hidden labels  [arxiv:2011.14027 §3.4]
    return bce(logits[hidden], labels[hidden]).mean()

The third covers iterative decoding for discrete tokens, following the MoMask base layer 2312.00063 §3.4. Masked diffusion differs in one rule: it unmasks at random with the schedule's probability instead of re-masking the least confident tokens 2406.07524 §4.1.

# Iterative masked decoding of the base layer  [arxiv:2312.00063 §3.4]
# The paper runs L = 10 iterations  [arxiv:2312.00063 §4]
def generate_base_tokens(text, length, steps):
    # Start with every base-layer position masked  [arxiv:2312.00063 §3.4]
    tokens = full(length, MASK_ID)                 # (n,)
    for step in range(steps):
        # Predict all masked positions at once  [arxiv:2312.00063 §3.4]
        logits = masked_transformer_cfg(tokens, text)     # (n, K)
        sampled = sample_categorical(logits)               # (n,)
        # Keep the new samples only at masked positions  [arxiv:2312.00063 §3.4]
        is_masked = tokens == MASK_ID
        tokens = where(is_masked, sampled, tokens)
        # Confidence of each new sample; fixed tokens are never re-masked  [arxiv:2312.00063 §3.4]
        confidence = where(is_masked, prob_of(logits, sampled), INFINITY)
        # Re-mask the least confident tokens, cosine schedule  [arxiv:2312.00063 §3.4]
        count = ceil(cos(pi * (step + 1) / (2 * steps)) * length)
        lowest = argsort(confidence)[:count]
        tokens[lowest] = MASK_ID
    return tokens

The fourth is the masked-diffusion denoising loss on text 2406.07524 §3.5 2406.07524 app. D.2. Its weight is 1/t under the log-linear schedule, as noted above.

# One masked-diffusion training step on text  [arxiv:2406.07524 alg. 1]
# Input: tokens (B, L) of token ids; pad_mask (B, L), 1 for real tokens
def mdlm_training_step(tokens, pad_mask):
    # Draw one masking time per sequence  [arxiv:2406.07524 app. D.3]
    times = uniform(B, low=eps, high=1.0)              # (B,)
    # Mask each token independently with probability t  [arxiv:2406.07524 app. D.2]
    is_masked = uniform(B, L) < times[:, None]         # (B, L)
    noisy = where(is_masked, MASK_ID, tokens)          # (B, L)
    # Log-probability of the clean token; 0 at unmasked  [arxiv:2406.07524 §3.2.3]
    log_probs = subs(backbone(noisy, times), noisy)    # (B, L, V)
    true_logp = gather(log_probs, tokens)              # (B, L)
    # Weight 1/t from eq. 10 under the log-linear schedule  [arxiv:2406.07524 §3.5]
    per_token = -(1.0 / times[:, None]) * true_logp    # (B, L)
    return sum(per_token * pad_mask) / sum(pad_mask)   # token average

The fifth covers two-segment pretraining with a relation head, as in LinkBERT. It adds one term to the first sketch 2203.15827 §4.2 [arxiv:2203.15827 eq. 1].

# MLM plus a relation loss on [CLS], as in LinkBERT  [arxiv:2203.15827 §4.2]
# Input: tokens (B, n) for [CLS] X_A [SEP] X_B [SEP]; relations (B,) in {0,1,2}
def linkbert_loss(model, tokens, relations):
    # Mask 15% of tokens: 80% [MASK], 10% random, 10% kept  [arxiv:2203.15827 §3]
    masked, targets = apply_mlm_masking(tokens)
    hidden = model.encode(masked)                           # (B, n, d)
    mlm = masked_cross_entropy(model.mlm_head(hidden), targets)
    # Three-way relation head on [CLS], replacing NSP  [arxiv:2203.15827 §4.2]
    drp = cross_entropy(model.drp_head(hidden[:, 0]), relations)  # (B, 3)
    # Equal weights, 1:1  [arxiv:2203.15827 eq. 1]
    return mlm + drp

Where the papers leave this open:

• Dynamic or static masks. CamemBERT redraws masks each time a sequence is used 1911.03894 §4.3. FlauBERT also uses dynamic masking 1912.05372 tab. 1. The first sketch above does not say which to use.

• Draw per unit or per token. For whole-word masking, the 80/10/10 draw is not stated as per word or per subword piece 1911.03894 §4.3. The sketch draws per unit, which is our choice.

• Random replacement. CamemBERT says "a random token" without saying how it is drawn 1911.03894 §4.3.

• Masking rate in FlauBERT. The text does not state it 1912.05372 tab. 1. BEHRT and CamemBERT both use 15% 1907.09538 §2.4 1911.03894 §4.3.

• Distribution of the hidden-label count. The card gives the range for label masking but does not restate the distribution 2011.14027 §3.4.

• Mask-ratio rounding and replacement split in MoMask. The paper writes a ceiling for the mask count and an 80/10/10 split 2312.00063 §3.2. The released code rounds the count with a minimum of 1, and applies 10% random tokens and then 88% of the rest as masks, which is about 79/10/11 2312.00063 §3.2 transformer.py.

• Re-mask schedule in MoMask. The paper indexes the schedule by iteration count 2312.00063 §3.4. The released code maps steps onto an evenly spaced grid from 0 to 1, so the step-to-ratio mapping differs transformer.py. The sketch uses the paper's text.

• Noise type in TSDAE. The text names deletion and swapping as noise 2104.06979 §3.1, but the released code implements only deletion denoising_auto_encoder.py. The paper's comparison of five noise types is the only evidence for the others 2104.06979 app. A 2104.06979 tab. 5.

• Masking scope in GraphFormers. The paper masks tokens of every input node 2105.02605 §3.3. The released code masks only the centre node data_handler.py.

• Replacement rule in DRAGON. The paper says only that a fraction of tokens is masked 2210.09338 §2.3. The released code uses whole-word masking by default, with 80% mask, 10% random and 10% kept dragon.py data_utils.py.

• Masking unit and rate in GottBERT. The paper gives neither, and the card says not to fill the gap with RoBERTa defaults unless you label them as your own assumptions 2012.02110 §3 2012.02110 §5.

• Masking in data2vec text. The paper's main text uses BERT masking, while the released config uses span masking, and the card places the config with a GLUE row rather than the main row 2202.03555 §4.3 2202.03555 tab. 4 base.yaml.

• Special tokens in ALBEF. The paper does not say whether [CLS] or [SEP] may be masked. The released code protects only padding and [CLS], so [SEP] can be masked model_pretrain.py.

• Head sharing in TabTransformer-MLM. The paper describes a per-column classifier for the replaced-value variant, but says only "a multi-class classifier" for MLM 2012.06678 §2 2012.06678 app. A.

• Averaging in StereoSet's masked score. The card records that the paper averages log probabilities and the code averages raw probabilities 2004.09456 §7.1 eval_discriminative_models.py.

• Masking rate in BETO. The paper gives neither the masking rate nor the replacement rule 2308.02976 §3.

• Replacement rule in TaBERT. The paper does not state the 80/10/10 split for context masking 2005.08314 §3.2.

• Link sampling in LinkBERT. The paper does not state the segment length, how a segment is picked from a linked document, or what an anchor with no outgoing link does 2203.15827 §4.2 2203.15827 §4.3. The 1/in-degree normalisation is inferred, not stated.

• Masking rate in the ad-headline model. The card states it is not given 2607.06818 §3.1. Its sampled-token stage also has no stated normalisation for the summed log-probability 2607.06818 eq. 6.

• Masking rate in FLASH's C4 MLM runs. The card does not state it, and no code is released 2202.10447 tab. 6.

• Relation head in LinkBERT base and large. These models start from BERT checkpoints whose head is next-sentence prediction. The paper does not say whether the three-class head starts fresh 2203.15827 §5.1.

Contradictions

Within the cards, the masking rules mostly agree. CamemBERT, FlauBERT and BEHRT mask units of the same kind, as tokens or codes. The label method and the cloze method use different units or no masked training at all. TSDAE deletes words rather than masking them, and the GraphFormers warm-up uses masks without a recovery loss. MoMask and MDLM both mask discrete units, but their schedules are different, and the cards do not compare them. The metrics also differ: tagging, parsing, NER and XNLI accuracy in CamemBERT, FLUE scores in FlauBERT, average precision and AUROC in BEHRT, mAP in the label method, precision at one in the cloze method, FID and R-precision in MoMask, bounded perplexity in MDLM, balanced accuracy in the partisan study, accuracy in Belebele, ARLUE and GLUE scores in the Arabic and DeBERTaV3 cards, a macro average over 35 tasks in FLAVA, and LAMBADA accuracy in GLM-130B. Their numbers should not be compared directly.

CamemBERT's whole-word result and FlauBERT's subword setup are on different corpora and benchmarks, so the cards do not test the whole-word question for FlauBERT. Neither card settles it.

Inside CamemBERT, the NER result in tab. 8 is higher for subword masking (91.55 vs 91.44), while the authors summarise the masking effect as small. The number 91.55 also appears for a different NER setting in tab. 5, which the card flags as an internal inconsistency. The CCNet BASE whole-word 100k model also appears with two different UPOS/LAS pairs in tab. 8 1911.03894 tab. 8. Check the tables before quoting these values.

TSDAE's advantage over MLM is not uniform. In the setting that trains on the target domain and then NLI+STS, the unsupervised averages are 56.5 for TSDAE and 55.9 for MLM, and MLM is ahead on AskUbuntu, 60.6 against 59.4 2104.06979 tab. 2. The paper reports TSDAE ahead of MLM as a pretraining step on most datasets, with AskUbuntu as the exception 2104.06979 §6.

For the cloze method, the main text says the headline results tune only layer 0. Table 1 matches the all-layer variant instead 2104.06599 §4.7 2104.06599 tab. 5.

Inside MoMask, the HumanML3D FID for the full model is 0.045 in the main table 2312.00063 tab. 1 and 0.051 in the ablation table [arxiv:2312.00063 tab. 2]. The KIT-ML value is also different across those tables. Cite the table you use.

Inside MDLM, the text says the model approaches autoregressive perplexity within 14% on LM1B 2406.07524 §5.1. Against the table, the 327B-token bound is about 10% above the autoregressive baseline, and the 33B-token bound about 21% above 2406.07524 tab. 1. Recompute before quoting the percentage.

TabTransformer's headline margin is measured with MLM. At 50 labels on the 8 larger datasets, MLM scores 1.2 points above the best other method, while the replaced-value variant scores 1.0 points above it 2012.06678 tab. 3. On the 12 smaller datasets, MLM trails the best other method at 50 and 200 labels 2012.06678 tab. 4. Read the margin claim as the better of the two objectives, not as a claim about MLM alone.

BAE's headline claim is that its attacks lower accuracy more than TextFooler in most settings 2004.01970 §3. The paper's own tables disagree for one case: with BERT as the target and replacement only, TextFooler gives lower accuracy than BAE-R on six of seven datasets 2004.01970 tab. 1 2004.01970 tab. 2. The text says TextFooler is sometimes stronger on BERT 2004.01970 §3. Use the tables for BERT targets.

Two paper and code pairs disagree on the masking rule. data2vec's paper uses BERT masking in its main text, while the text config uses spans with no random or unmasked tokens 2202.03555 §4.3 base.yaml. StereoSet's paper averages log probabilities for the masked score, and its code averages raw probabilities 2004.09456 §7.1 eval_discriminative_models.py.

BERTje's paper and released code disagree on the masking unit and rate. The paper hides consecutive wordpieces of the same word, at 15% of tokens 1912.09582 §2.2. The code keeps each wordpiece as a separate candidate, and caps predictions at 20 per sequence, so a full-length text is masked at about 4% create-pretrain-data.py create-pretrain-data.py. The card's advice is to decide which rule you mean before you train.

ChemBERTa-2's comparison of MLM and MTR is on SMILES strings, not text. It does not test the text variants above. MTR is ahead on all eight Table 1 columns at 10M and 77M, but on six of eight at 5M, where MLM is ahead on BACE RMSE and BACE ROC 2209.01712 tab. 1.

Results from the new cards are not directly comparable with the rest. The RoBERTa gain of +3 to +4 points is for a bundle of changes [arxiv:2103.11943 p. 12]. LinkBERT's MRQA scores use a dev split that the authors divided in two, so they are not leaderboard numbers 2203.15827 §5.2. ARBERT's ARLUE scores are on test splits 2101.01785 §5.2, while DeBERTaV3's GLUE numbers are dev-set results, with the fine-tuning grid chosen on dev 2111.09543 §6.4. The ad-headline ROUGE-L gains are on proprietary campaign data, against the paper's own bi-LSTM baseline 2607.06818 §4.2. FLASH's speed-ups are training-cost comparisons on TPU-v4, against Transformer baselines that the authors re-ran with their own code 2202.10447 §4.2.

Papers

• Ad Headline Generation using Self-Critical Masked Language Model: BERT-Large trained with masked-token loss on headline tokens, then tuned with self-critical policy gradient and ROUGE-L as reward.
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation: a fusion encoder with MLM and momentum distillation, after a contrastive alignment step.
• ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic: two BERT-Base encoders for Arabic, one on formal text and one on tweets, with the ARLUE benchmark.
• BAE: BERT-based Adversarial Examples for Text Classification: black-box word substitutions and insertions proposed by BERT's masked LM, filtered by sentence similarity and part of speech.
• BEHRT: Transformer for Electronic Health Records: a BERT-style Transformer over diagnosis histories, pretrained with masked disease prediction.
• BERT: A Review of Applications in Natural Language Processing and Understanding: survey of BERT's pretraining and uses, with MLM background, dynamic masking and translation language modeling. Its figures are taken from cited papers.
• BERTje: A Dutch BERT Model: BERT-base trained on Dutch text only, with sentence order prediction and a masking rule that the paper and code state differently.
• BOOM: a benchmark that holds out low-density molecules for property prediction; cited here as a weak link only.
• CamemBERT: a Tasty French Language Model: RoBERTa-style French encoder trained with whole-word masking; the ablation isolates masking granularity.
• ChemBERTa-2: Towards Chemical Foundation Models: RoBERTa-style encoder over SMILES, pretrained with MLM or with regression on RDKit descriptors, compared on MoleculeNet.
• data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language: one self-distillation recipe for speech, images and text, with a masked student that regresses teacher features.
• DeBERTaV3: ELECTRA-style pretraining with gradient-disentangled embedding sharing: replaces the masked-token loss with replaced-token detection, and shares embeddings with a stop-gradient on the generator side.
• Deep Bidirectional Language-Knowledge Graph Pretraining: an encoder pretrained with MLM and graph link prediction on text and a local knowledge-graph subgraph.
• Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion: per-token noise levels for sequence models; cited for its framing of masking as partial noise.
• Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora: shared-task report whose grammar evaluation scores masked models by pseudo-likelihood.
• FlauBERT: Unsupervised Language Model Pre-training for French: BERT-style French encoder with subword MLM on 71 GB of text, evaluated on the FLUE benchmark.
• FLAVA: A Foundational Language And Vision Alignment Model: image, text and fusion encoders trained with masked multimodal, masked image, masked language and contrastive losses on public image-text pairs.
• From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models: probes encoder and decoder models for political lean, and measures how partisan pretraining changes hate-speech and misinformation classifiers.
• General Multi-label Image Classification with Transformers: hides labels in a Transformer classifier and predicts them, so known labels can be given at test time.
• GLM-130B: An Open Bilingual Pre-trained Model: a 130B-parameter English and Chinese model trained by blank infilling, with stability fixes and INT4 weight quantization.
• GottBERT: a pure German Language Model: RoBERTa-style German encoder pretrained on filtered and unfiltered German OSCAR.
• GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph: nests graph attention inside a BERT-style encoder, with a masked warm-up stage before fine-tuning.
• Having Beer after Prayer? Measuring Cultural Bias in Large Language Models: Arabic prompts with masked entities, scored by a cultural bias measure.
• Learning How to Ask: Querying LMs with Mixtures of Soft Prompts: learns continuous cloze prompts for querying a fixed masked language model.
• LinkBERT: Pretraining Language Models with Document Links: MLM on two-segment inputs built from document links, with a three-way relation loss in place of next-sentence prediction.
• MoMask: Generative Masked Modeling of 3D Human Motions: masked transformer over the base layer of a residual quantizer, then a residual transformer, for text-to-motion generation.
• Simple and Effective Masked Diffusion Language Models: trains an encoder to recover masked tokens under a weighted masked-token loss, and samples text by unmasking from an all-mask start.
• Spam and Sentiment Detection in Arabic Tweets Using MARBERT Model: fine-tunes the MARBERT encoder for five-class sentiment and for spam on private customer tweets, with class imbalance as the main difficulty.
• Spanish Pre-trained BERT Model and Evaluation Data: BERT-Base-sized encoder pretrained on Spanish text only, with whole-word and dynamic masking, and a Spanish evaluation suite.
• StereoSet: Measuring stereotypical bias in pretrained language models: context association tests with a stereotype, an anti-stereotype and an unrelated option, scored on language modeling and stereotype preference.
• TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data: BERT-initialised encoder over sampled table rows, pretrained with MLM and masked column and cell objectives.
• TabTransformer: Tabular Data Modeling Using Contextual Embeddings: Transformer over categorical columns, with masked or replaced-value pretraining for semi-supervised use.
• The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants: a multiple-choice reading test, parallel across 122 variants, used to evaluate masked encoders and LLMs.
• Transformer Quality in Linear Time: gated attention unit with chunked exact and linear attention, evaluated on C4 MLM and LM at long context.
• TSDAE: sentence embeddings from deletion-based reconstruction, compared with MLM as an unsupervised pretraining step.