--- kind: concept slug: masked-language-modeling title: Masked language modeling summary: >- Hide part of an input and train a model to recover it from the visible rest, with loss at the hidden positions. Covers encoder pretraining in several languages, table-and-text, tabular, fusion and label masking, cloze prompts, masked diffusion, iterative decoding, replaced-token detection, blank infilling, adversarial fill-ins and bias probes. The unit you hide is the main design choice. tags: [pretraining, encoder, self-supervised, masking, transformer, generation, diffusion, sentence-embeddings, knowledge-graph, bias, tabular, vision-language, tables, adversarial, document-links, cross-lingual, arabic, replaced-token-detection, blank-infilling, headline-generation, reinforcement-learning, long-context] papers: ["1907.09538", "1911.03894", "1912.05372", "1912.09582", "2004.01970", "2004.09456", "2005.08314", "2011.14027", "2012.02110", "2012.06678", "2101.01785", "2103.11943", "2104.06599", "2104.06979", "2105.02605", "2107.07651", "2111.09543", "2112.04482", "2202.03555", "2202.10447", "2203.15827", "2209.01712", "2210.02414", "2210.09338", "2305.08283", "2305.14456", "2308.02976", "2308.16884", "2312.00063", "2406.07524", "2407.01392", "2504.08165", "2505.01912", "2606.25495", "2607.06818"] related: [language-model-pretraining, permutation-language-modeling, dataset-curation, prompt-engineering, self-attention, discrete-diffusion, diffusion-models, llm-evaluation, denoising-autoencoder, text-embedding, graph-neural-network, knowledge-graph-completion, multilingual-evaluation, llm-ethics, algorithmic-fairness, self-supervised-learning, masked-image-modeling, vision-language-pretraining, semi-supervised-learning, knowledge-distillation, adversarial-text-generation, table-question-answering, text-classification, extractive-question-answering, multi-hop-reasoning, transfer-learning, replaced-token-detection, subword-tokenization, weight-quantization, sentiment-analysis, linear-attention, policy-gradient, sequence-labeling, molecular-representation-learning] status: draft updated_at: 2026-11-01 --- ## Overview Masked language modeling (MLM) hides some units of an input and trains a model to recover the hidden units from the visible ones. The loss is taken only at the hidden positions [arxiv:1911.03894 §4.3]. The cards here use the same idea on different inputs and for different goals: - **Text pretraining.** An encoder is pretrained by recovering masked word pieces, here in French [arxiv:1911.03894 §4.3] [arxiv:1912.05372 §3.2], or trained on German text only [arxiv:2012.02110 §3]. - **Arabic text.** One encoder is pretrained on formal Arabic with whole-word masking, and another on Arabic tweets without the next-sentence objective [arxiv:2101.01785 §3.1] [arxiv:2101.01785 §3.2]. - **Dutch text.** A BERT-base encoder is pretrained on Dutch text alone, with sentence order prediction in place of next-sentence prediction. The paper describes whole-word masking, but the released code switches it off [arxiv:1912.09582 §2.2] [code: pretraining/create-pretrain-data.py#L32]. - **Molecules.** A RoBERTa-style encoder reads SMILES strings. It is pretrained with MLM, or with regression on about 200 computed properties, which the authors compare directly [arxiv:2209.01712 §3.1] [arxiv:2209.01712 tab. 1]. - **Linked documents.** An encoder is pretrained on two-segment inputs. The second segment is the next one from the same document, a random one, or one from a linked document. A three-way relation head is trained next to MLM [arxiv:2203.15827 §4.2]. - **Parallel text.** Translation language modeling masks words in a pair of parallel sentences, and the model may use either sentence to fill a gap [arxiv:2103.11943 p. 9]. - **Clinical records.** A Transformer over patient histories is pretrained by recovering masked diagnosis codes [arxiv:1907.09538 §2.4]. - **Tabular data.** A Transformer over categorical columns is pretrained by recovering masked feature values [arxiv:2012.06678 §2]. - **Table and text.** A BERT-initialised encoder reads a question with sampled table rows, and is pretrained with MLM on the text plus masked column and cell objectives [arxiv:2005.08314 §3.2]. - **Vision-language.** A fusion encoder that reads caption tokens and image features is trained with MLM, optionally with a teacher's soft targets [arxiv:2107.07651 §3.2] [arxiv:2107.07651 §3.3]. - **Self-distillation.** A student regresses teacher representations at masked positions, with text as one of three modalities [arxiv:2202.03555 §3.3]. - **Labels.** A multi-label image classifier hides some of its label inputs and predicts them [arxiv:2011.14027 §3.4]. - **Querying.** A fixed masked model is not trained further. Instead a cloze-style prompt with a blank is learned, so the model fills in the blank with a fact [arxiv:2104.06599 §3]. - **Probing.** A masked encoder fills a blank in a fixed sentence, and the fill-in words are scored against a stance lexicon [arxiv:2305.08283 §2.1]. - **Generation.** A masked model starts from an all-blank sequence and fills blanks over several rounds. The cards use this for discrete motion tokens [arxiv:2312.00063 §3.4] and, as masked diffusion, for text [arxiv:2406.07524 §4.1]. - **Evaluation.** Masked encoders are fine-tuned with a multiple-choice head and compared with decoder models on a reading benchmark [arxiv:2308.16884 §4.2]. - **Sentence embeddings.** Words are deleted from a sentence, and a decoder rebuilds the sentence from one encoder vector. The paper compares this with MLM as an unsupervised pretraining step [arxiv:2104.06979 §3.1] [arxiv:2104.06979 §5.2]. - **Replaced-token detection.** A small generator fills masked slots, and a larger discriminator labels every token as original or replaced. The generator is trained with MLM; the discriminator takes the place of the masked-token loss [arxiv:2111.09543 §3.1]. - **Blank infilling.** A bidirectional model hides spans of its input, and writes the spans back. Most training sequences keep a prefix and hide a tail [arxiv:2210.02414 §2.1] [arxiv:2210.02414 §2.2]. - **Graph-text training.** A text-graph encoder hides spans of its inputs as augmentation during a warm-up stage [arxiv:2105.02605 §3.3]. A separate encoder is pretrained on text and a knowledge graph with MLM and link prediction together [arxiv:2210.09338 §2.3]. - **Bias measurement.** A masked model scores candidate entities in the blank of a culturally framed prompt [arxiv:2305.14456 §4.4]. A masked model also scores stereotype and anti-stereotype fill-ins in a sentence blank [arxiv:2004.09456 §7.1]. - **Scoring.** A masked model is scored by pseudo-likelihood on grammatical minimal pairs [arxiv:2504.08165 §5.2.1]. - **Adversarial text.** A masked model proposes replacement or inserted words that can flip a text classifier's label [arxiv:2004.01970 §2]. - **Vision-language with masked patches.** A fusion encoder predicts masked caption words and masked image-patch codes, with image-text matching on the same pass. Unimodal MLM and masked image modeling run on unpaired data [arxiv:2112.04482 §3.2] [arxiv:2112.04482 §3.3]. - **Ad headlines.** A BERT-Large model hides headline tokens during training, then is tuned with a reward on sampled headlines [arxiv:2607.06818 §3.1] [arxiv:2607.06818 §3.3]. - **Long context.** A linear-cost attention model is pretrained on C4 with MLM as one of its settings [arxiv:2202.10447 tab. 12]. The invariant across the MLM-style training variants is the scoring rule. The model is judged only on the hidden units, and the target is the original value [arxiv:1911.03894 §4.3] [arxiv:2210.09338 §2.3]. The masked diffusion and masked generation variants keep this rule: unmasked positions contribute nothing to the loss [arxiv:2406.07524 §3.5] [arxiv:2312.00063 §3.2]. The tabular, German and Spanish variants keep it too [arxiv:2012.06678 §2] [arxiv:2012.02110 §3] [arxiv:2308.02976 §3]. The table-and-text variant keeps MLM on the context and adds column and cell targets on top [arxiv:2005.08314 §3.2]. LinkBERT keeps the rule for its MLM term, and adds a relation loss on the [CLS] vector [arxiv:2203.15827 eq. 1]. Translation language modeling also keeps the rule, applied to a pair of parallel sentences [arxiv:2103.11943 p. 9]. ARBERT and MARBERT keep the rule for their masked-LM objective, with MARBERT dropping next-sentence prediction [arxiv:2101.01785 §3.1] [arxiv:2101.01785 §3.2]. The FLAVA masked multimodal loss keeps the rule, but the released code applies it only to image-text pairs that are matched [code: torchmultimodal/modules/losses/flava.py#L418-L437]. The GLM-130B loss is taken on the hidden spans [arxiv:2210.02414 §2.1]. Six variants depart from the rule. The sentence-embedding variant deletes tokens and rebuilds the whole sentence [arxiv:2104.06979 §3.1]. The graph warm-up uses a contrastive link loss on masked inputs [arxiv:2105.02605 §3.3]. data2vec regresses averaged teacher block outputs at the masked steps, not the token [arxiv:2202.03555 §3.3]. The momentum variant mixes the one-hot token with a teacher distribution [arxiv:2107.07651 §3.3] [arxiv:2107.07651 eq. 7]. The replaced-token discriminator scores every token, not only the masked ones [arxiv:2111.09543 §3.1]. The self-critical stage of the ad-headline model replaces the token target with a reward-weighted sampled token at the hidden positions [arxiv:2607.06818 eq. 9]. The querying, probing, bias, scoring and adversarial uses take the model without training on the hidden units. ## Variants ### Subword masking The baseline in the CamemBERT ablation. Individual subword tokens are the masking units [arxiv:1911.03894 §4.3]. In the authors' BASE model trained on OSCAR for 100k steps, it scores UPOS 97.78, LAS 89.80, NER 91.55 and XNLI 81.04 [arxiv:1911.03894 tab. 8]. Card: [CamemBERT: a Tasty French Language Model](/papers/1911.03894). FlauBERT also masks subword units, with masks redrawn dynamically. It pretrains with MLM only, on a 71 GB French corpus [arxiv:1912.05372 tab. 1] [arxiv:1912.05372 §3.1]. The masking rate is not stated in the text [arxiv:1912.05372 tab. 1]. Card: [FlauBERT: Unsupervised Language Model Pre-training for French](/papers/1912.05372). ### Whole-word masking Sample about 15% of the whitespace-delimited words, and hide all subword pieces of each sampled word. The authors also redraw the masks every time a sequence is used (dynamic masking), and drop next-sentence prediction [arxiv:1911.03894 §4.3]. Card: [CamemBERT: a Tasty French Language Model](/papers/1911.03894). In the same ablation, whole-word masking scores UPOS 97.79 and LAS 89.88 against 97.78 and 89.80 for subword masking. It scores XNLI 81.55 against 81.04, but NER 91.44 against 91.55 [arxiv:1911.03894 tab. 8]. The authors summarise the effect as +0.5 on XNLI and little elsewhere [arxiv:1911.03894 app. A]. Whole-word masking needs the mapping from subword pieces back to words, which the paper defines on whitespace in the untokenised text [arxiv:1911.03894 §4.3]. ### Dynamic masking in a RoBERTa-style recipe Dynamic masking draws a new mask each time a sequence is used. The original BERT used static masking [arxiv:2103.11943 p. 11]. The survey lists dynamic masking as one of RoBERTa's changes, with more text (about 160 GB from four corpora), no next-sentence prediction and larger batches [arxiv:2103.11943 p. 11–12]. Those changes together are reported as a gain of +3 to +4 points over BERT. The survey does not isolate the masking change, and it does not state the masking rate [arxiv:2103.11943 p. 11–12]. The RoBERTa paper has no card here, so these figures are the survey's account of it. ### Monolingual German encoder GottBERT is a RoBERTa-base or RoBERTa-large encoder trained only on German text, with fairseq [arxiv:2012.02110 §3]. Its pretraining is a RoBERTa-style masked LM [arxiv:2012.02110 §3]. The paper gives no masking rate and does not say whether masking is per subword or per word. Whole-word masking appears only as future work [arxiv:2012.02110 §5] [arxiv:2012.02110 §6]. Pretraining runs 100k updates with batch size 8k and a peak learning rate of 4e-4 for base [arxiv:2012.02110 §3]. On CoNLL 03 the best filtered base model scores 86.14 F1, against 85.37 for GELECTRA base [arxiv:2012.02110 tab. 3]. The paper reports single runs, so differences of about one point are within unknown noise [arxiv:2012.02110 §5]. Card: [GottBERT: a pure German Language Model](/papers/2012.02110). ### Monolingual Spanish encoder with whole-word and dynamic masking BETO is a BERT-Base-shaped encoder with 12 layers, hidden size 768 and 110M parameters, pretrained with MLM on Spanish text only [arxiv:2308.02976 §3]. It trains a cased and an uncased model [arxiv:2308.02976 §3]. Masking is whole-word: when one sub-word of a word is hidden, its contiguous sub-words are hidden too. It is also dynamic, with 10 different masks per sentence [arxiv:2308.02976 §3]. The masking rate and the replacement rule are not stated [arxiv:2308.02976 §3]. The corpus is about 3 billion words from Spanish Wikipedia and OPUS sources, including subtitles and news [arxiv:2308.02976 §3]. Pretraining runs 2M steps. The first 900k use batch size 2048 and length 128, and the rest use batch size 256 and length 512. The peak learning rate is 1e-4, after 10,000 warm-up steps [arxiv:2308.02976 §3]. The authors also compile Spanish tasks into GLUES [arxiv:2308.02976 §4]. Author-reported, BETO-cased scores 82.01 on XNLI and 88.43 on NER, against 78.50 and 87.38 for the best multilingual BERT the authors cite. Uncased is ahead on PAWS-X (89.55 against 89.00) and MLDoc (96.12 against 95.70) [arxiv:2308.02976 tab. 1]. On one TAR-to-XQuAD setting, the best mBERT leads on F1, 77.60 against 77.56 [arxiv:2308.02976 tab. 2]. The text reports 90.7 for mBERT trained on English PAWS-X data, a different setting from the 89.00 in Table 1 [arxiv:2308.02976 §5.2] [arxiv:2308.02976 tab. 1]. The paper does not say whether its scores are dev or test scores [arxiv:2308.02976 §5.2]. The corpus draws on OPUS sources with mixed licences, so check those terms before commercial reuse of the weights or the corpus [arxiv:2308.02976 §3]. Card: [Spanish Pre-trained BERT Model and Evaluation Data](/papers/2308.02976). ### Monolingual Arabic encoders: formal text and tweets ARBERT and MARBERT are BERT-Base-shaped encoders with the same architecture, differing in data and objective [arxiv:2101.01785 §3.1] [arxiv:2101.01785 §3.2]. ARBERT is pretrained on about 61 GB of mostly formal Arabic, with whole-word masking at 15%, 80/10/10 [arxiv:2101.01785 §3.1]. MARBERT is pretrained on 1B sampled tweets, with next-sentence prediction dropped and its masking details not restated [arxiv:2101.01785 §3.2]. Both use a 100K WordPiece vocabulary [arxiv:2101.01785 §3.1] [arxiv:2101.01785 §3.2]. Author-reported, on the test splits of the ARLUE benchmark, the strongest entry is MARBERT-v2, at 77.40 against 76.55 for XLM-R Large [arxiv:2101.01785 §5.2]. The paper never defines MARBERT-v2 beyond the label, so treat it as a name only [arxiv:2101.01785 §7]. The MARBERT tweet data is in-house and will not be released, and the checkpoints are for research use only [arxiv:2101.01785 §3.2]. Card: [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](/papers/2101.01785). ### Monolingual Dutch encoder with sentence order prediction BERTje is a BERT-base model trained on about 12 GB (about 2.4B tokens) of Dutch books, news, Wikipedia and SoNaR text [arxiv:1912.09582 §2.1]. Each training pair is either in order or swapped, and the swapped case is the sentence-order task that replaces next-sentence prediction [arxiv:1912.09582 §2.2]. Its masked-LM rule is 15% of tokens with 80/10/10. The paper says the hidden tokens are consecutive wordpieces of the same word, so whole words are masked together [arxiv:1912.09582 §2.2]. The released code turns whole-word masking off, and caps predictions at 20 per sequence, so a full-length text gets a masked share of about 4%, not 15% [code: pretraining/create-pretrain-data.py#L32] [code: pretraining/create-pretrain-data.py#L34]. The released run command sets 1M steps, batch size 256, peak learning rate 1e-4 and 10,000 warm-up steps [code: pretraining/run-pretraining.sh#L12-L16]. The paper gives none of these. Author-reported, on the CoNLL-2002 test split, BERTje scores 88.3 F1 against 80.7 for multilingual BERT [arxiv:1912.09582 tab. 1]. On UD-LassySmall test accuracy it scores 96.3 against 92.5 [arxiv:1912.09582 tab. 2], and on the 110k Dutch book-review test set 93.0 against 89.1 [arxiv:1912.09582 tab. 4]. The paper reports single runs, so gaps of about a point are not established [arxiv:1912.09582 §4]. The README reports 90.24 on the CoNLL-2002 test set, which differs from the paper's 88.3 [code: README.md#L77]. Card: [BERTje: A Dutch BERT Model](/papers/1912.09582). ### Translation language modeling across parallel sentences Translation language modeling (TLM) is the cross-lingual objective from XLM (arXiv:1901.07291). It masks words in a pair of parallel sentences and lets the model use either sentence to fill a gap [arxiv:2103.11943 p. 9]. Without parallel text, the fallback is masked language modeling alone [arxiv:2103.11943 p. 10]. The survey reports average machine-translation gains over the best known models of +4.9% for TLM and +1.3% for MLM alone [arxiv:2103.11943 p. 10]. These figures are the survey's account of work it cites. The XLM paper has no card here, so check it on arXiv before quoting the numbers. ### Masked diagnosis codes in patient records Here the units are diagnosis codes in a patient's visit history. The method follows BERT: 15% of the tokens are selected, and of those, 80% become MASK, 10% a random vocabulary token, and 10% stay unchanged. The loss is computed only at the selected positions [arxiv:1907.09538 §2.4]. This pretraining is followed by a fine-tuning stage that predicts diagnoses at the next visit or within 6 and 12 months [arxiv:1907.09538 §2.5]. The paper reports pretraining precision at threshold 0.5, 0.6597 for the tuned architecture [arxiv:1907.09538 §2.4] [arxiv:1907.09538 tab. 2]. Its downstream comparison against DeepR and RETAIN is on UK primary-care data, and the authors report average precision and AUROC on that comparison [arxiv:1907.09538 tab. 1]. Card: [BEHRT: Transformer for Electronic Health Records](/papers/1907.09538). ### Masked feature prediction in tabular data Here the units are the categorical features of one row. The method picks k% of them, here k = 30, and sets each to a missing-value embedding. A multi-class classifier then predicts the original class of each masked feature from its contextual embedding [arxiv:2012.06678 §2]. The paper does not say whether that classifier is shared across columns [arxiv:2012.06678 §2]. This pretraining is used only in semi-supervised runs [arxiv:2012.06678 §3]. On the 8 datasets with more than 30K rows, the MLM variant scores 66.8, 71.0 and 72.9 mean AUC at 50, 200 and 500 labels, against 66.6, 70.9 and 73.1 for the replaced-value variant [arxiv:2012.06678 tab. 3]. On the 12 smaller datasets, MLM scores 78.5, 81.0 and 82.4, against 78.6, 81.6 and 83.4 for the replaced-value variant. The best other method at each label count scores 79.4, 81.2 and 82.3, and the paper names a different method at each count [arxiv:2012.06678 tab. 4]. Card: [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](/papers/2012.06678). ### Masked context and column objectives for table-and-text pretraining TaBERT encodes an utterance with K rows of a table. Each row is one BERT input: the utterance tokens, then one string per cell holding the column name, a type and the cell value [arxiv:2005.08314 §3.1]. It starts from uncased BERT-Base or BERT-Large [arxiv:2005.08314 §5]. Pretraining sums three losses [arxiv:2005.08314 app. A.2]. The first is MLM on the context, at 15% of sub-tokens [arxiv:2005.08314 §3.2]. The second is masked column prediction: 20% of columns have their names and types hidden in every row, and the model predicts the hidden tokens from the column vector [arxiv:2005.08314 §3.2]. The third is cell value recovery, which predicts the value tokens of the hidden columns from each cell's vector with a span head. Cell values stay visible in the input [arxiv:2005.08314 §3.2]. The paper does not state the replacement rule for context masking [arxiv:2005.08314 §3.2]. Training data is 26.6M examples, 1.3M from Wikipedia tables and 25.3M from the WDC WebTable Corpus, each with nearby text [arxiv:2005.08314 §3.2] [arxiv:2005.08314 app. A.1]. The objective ablation on TaBERT-Base with K = 3 compares MCP alone with MCP plus cell value recovery. On WikiTableQuestions dev the scores are 51.6 ± 0.7 and 51.6 ± 0.5. On Spider dev they are 62.6 ± 0.7 and 63.3 ± 0.7. The authors call the cell-value gain slight [arxiv:2005.08314 tab. 5] [arxiv:2005.08314 §5]. Author-reported, MAPO with TaBERT-Large and K = 3 reaches 52.3% test accuracy for the run that was best on dev, against 46.9% for the earlier ensemble of 10 [arxiv:2005.08314 tab. 1]. The mean over runs is 51.8 ± 0.6 [arxiv:2005.08314 tab. 1]. The authors say some WikiTableQuestions test tables may be in the pretraining corpus, but judge the share tiny [arxiv:2005.08314 §5]. The largest model took about six days on 128 V100 GPUs [arxiv:2005.08314 app. A.2]. The released code is non-commercial, so this section relies on the paper alone. Card: [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](/papers/2005.08314). ### Label masking for multi-label classification Here the units are labels rather than tokens. For each training image, the method picks how many labels to hide: a number between 0.25 and 1 times the number of labels [arxiv:2011.14027 §3.4]. Hidden labels get an "unknown" state in the input, and the other labels keep their ground-truth state. The loss is binary cross-entropy on the hidden labels only [arxiv:2011.14027 §3.4]. The paper justifies the lower bound by comparison with typical MLM masking rates, and by the assumption that inputs with more than 75% of labels known are unrealistic [arxiv:2011.14027 §3.4]. Because the model learns to accept known states, labels known at test time can be fed in for a single forward pass [arxiv:2011.14027 §4.2]. The authors report 85.1 mAP on COCO-80 [arxiv:2011.14027 tab. 1]. Card: [General Multi-label Image Classification with Transformers](/papers/2011.14027). Before reproducing this, read the card's Paper vs code section. The released loader switches label masking off for two of the datasets [code: dataloaders/data_utils.py#L17-L23]. ### Masked span warm-up for textual graphs GraphFormers nests a graph-attention step in each layer of a BERT-style encoder, so a node's text is encoded together with its neighbours' text. Its training has two stages [arxiv:2105.02605 §3.3]. The warm-up stage runs on inputs where 15% of tokens are masked in spans, with the 80/10/10 rule [arxiv:2105.02605 app. A.1]. The loss in that stage is the contrastive link-prediction loss of [arxiv:2105.02605 eq. 7], not recovery of the masked tokens. The masks therefore act as data augmentation. The fine-tuning stage then runs on unmasked data. Without the progressive stage, the paper reports P@1 drops of 0.98, 1.71 and 1.18 points on Product, DBLP and Wiki [arxiv:2105.02605 §4.4] [arxiv:2105.02605 tab. 4]. Card: [GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph](/papers/2105.02605). ### Joint masked and graph link prediction DRAGON pretrains one encoder on a text segment and a local knowledge-graph subgraph. Two losses are summed with weight 1 each. The first is MLM on the text, with cross-entropy at the masked positions. The second is link prediction on held-out graph edges [arxiv:2210.09338 §2.3] [arxiv:2210.09338 eq. 8]. The pretraining masks 15% of text tokens and holds out 15% of edges [arxiv:2210.09338 §3.1]. In the authors' ablation, MLM alone scores 74.3 on CSQA and 67.2 on OBQA. Link prediction alone scores 73.8 and 66.4. The joint objective scores 76.0 and 72.0 [arxiv:2210.09338 tab. 5]. The general model took 7 days on eight A100 GPUs in FP16 [arxiv:2210.09338 §3.1]. The "RoBERTa" baseline in the paper is RoBERTa-Large continued with MLM on the same text and steps, not the released checkpoint [arxiv:2210.09338 §3.3]. Card: [Deep Bidirectional Language-Knowledge Graph Pretraining](/papers/2210.09338). ### Two-segment inputs with document links LinkBERT builds a graph of documents from hyperlinks, or from citations in biomedical papers [arxiv:2203.15827 §4.1]. Each training input holds an anchor segment and a second segment. The second segment is the next one of the same document, a random one, or one from a document the anchor links to. The three options are equally likely in the main setting [arxiv:2203.15827 §4.2] [arxiv:2203.15827 §5.1]. A linked document is drawn with probability inversely proportional to its in-degree [arxiv:2203.15827 §4.3]. The model keeps MLM at 15% with the 80/10/10 rule [arxiv:2203.15827 §3]. It adds a three-way relation head on the [CLS] vector, which replaces next-sentence prediction, and the two losses have weight 1:1 [arxiv:2203.15827 §4.2] [arxiv:2203.15827 eq. 1]. Author-reported, MRQA average F1 is 77.8 for LinkBERT-base against 75.2 for BERT-base, and 81.0 against 78.5 for large [arxiv:2203.15827 tab. 1]. Without the relation loss, base scores fall on the datasets reported, for example HotpotQA from 78.2 to 76.5 [arxiv:2203.15827 tab. 6]. The authors state that a random-link tiny model matches BERT-tiny [arxiv:2203.15827 §5.5]. Base models took four days on four A100 GPUs, and the biomedical base took seven days on eight [arxiv:2203.15827 §5.1] [arxiv:2203.15827 §6.1]. The cost of building the graph is not stated [arxiv:2203.15827 §6.1]. The released repository holds fine-tuning and preprocessing scripts, not the pretraining loop, so the link sampling and relation head cannot be checked against code. Card: [LinkBERT: Pretraining Language Models with Document Links](/papers/2203.15827). ### Masked fusion with momentum distillation ALBEF first aligns image and text features with a contrastive loss [arxiv:2107.07651 §3.2], then runs a fusion encoder in which the text tokens attend to image tokens [arxiv:2107.07651 §3.1]. Its MLM loss selects 15% of caption tokens, with the 80/10/10 rule [arxiv:2107.07651 §3.2]. The target is the one-hot vocabulary label [arxiv:2107.07651 eq. 3]. The momentum variant mixes this loss with a KL term to an EMA teacher's distribution over the vocabulary, at each masked position, with weight α = 0.4 [arxiv:2107.07651 eq. 7] [arxiv:2107.07651 §3.5]. On 4M pretraining images, the momentum term on MLM raises VQA test-dev from 74.06 to 74.42 and NLVR² test-P from 80.34 to 80.44 [arxiv:2107.07651 tab. 1]. The card notes that the EMA copy doubles weight memory, because the teacher's encoders run in a second forward pass each step [code: models/model_pretrain.py#L63-L76]. Card: [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](/papers/2107.07651). ### Sentence reconstruction by token deletion TSDAE is not masking in the BERT sense. It deletes words from a sentence and trains a decoder to rebuild the sentence from the encoder's single vector. The decoder's cross-attention sees only that vector [arxiv:2104.06979 §3.2]. The best deletion ratio in the appendix is 0.6, meaning 60% of tokens are affected [arxiv:2104.06979 app. A] [arxiv:2104.06979 tab. 6]. After training the decoder is discarded, and the encoder alone produces embeddings [arxiv:2104.06979 §3.2]. Over four domain tasks, the unsupervised average is 55.2 for TSDAE and 52.9 for MLM with BERT-base [arxiv:2104.06979 tab. 2]. In the appendix tuning run, deletion at ratio 0.3 scores 78.33 on STS-B and masking scores 76.56 [arxiv:2104.06979 app. A] [arxiv:2104.06979 tab. 5]. Card: [TSDAE](/papers/2104.06979). ### Replaced-token detection with gradient-disentangled embedding sharing DeBERTaV3 swaps DeBERTa's masked-token objective for ELECTRA-style replaced token detection [arxiv:2111.09543 §3.1]. A generator, half the discriminator's depth, is trained with MLM on about 15% of tokens. Its sampled replacements are fed to a discriminator that labels every token as original or replaced [arxiv:2111.09543 eq. 3] [arxiv:2111.09543 eq. 4]. The paper's own contribution is the embedding sharing rule. The discriminator's token embedding is the generator's with its gradient stopped, plus a residual that only the discriminator trains [arxiv:2111.09543 §3.3]. In a 125k-step ablation, vanilla sharing scores 88.8/88.4 on MNLI-m/mm, no sharing 88.3/87.9, and the gradient-disentangled version 89.3/89.0 [arxiv:2111.09543 tab. 2]. Author-reported, DeBERTaV3-Large reaches a GLUE dev average of 91.37, against 90.00 for DeBERTa-large and 89.46 for ELECTRA-large [arxiv:2111.09543 tab. 3]. The ablations use a smaller pretraining regime than the full runs, so the two sets of numbers are not comparable [arxiv:2111.09543 §4]. The paper weights the discriminator loss by λ = 50 [arxiv:2111.09543 §3.1], while the released code defaults to 10 [code: DeBERTa/apps/tasks/rtd_task.py#L392]. Card: [DeBERTaV3: ELECTRA-style pretraining with gradient-disentangled embedding sharing](/papers/2111.09543). ### Masked multimodal and unimodal objectives on public pairs FLAVA has an image encoder, a text encoder and a six-layer multimodal encoder [arxiv:2112.04482 §3.1]. On image-text pairs, its masked multimodal loss masks rectangular blocks of image patches, tokenised by a pretrained dVAE, and 15% of text tokens. The multimodal encoder predicts both sets of masked targets [arxiv:2112.04482 §3.2]. On unpaired text it runs MLM at 15%, and on unpaired images masked image modeling on dVAE tokens [arxiv:2112.04482 §3.3]. The text encoder starts from MLM on CCNews and BookCorpus, and the image encoder from DINO [arxiv:2112.04482 §3.3] [arxiv:2112.04482 app. C]. Author-reported, the full model reaches a macro average of 75.85 over 35 tasks, against 66.78 for CLIP ViT-B/16 trained on 400M pairs [arxiv:2112.04482 tab. 3] [arxiv:2112.04482 tab. 4]. Keeping only the multimodal losses gives 74.23, and removing the unimodal initialisation gives 73.72 [arxiv:2112.04482 tab. 3]. The column order in table 3 is wrong for some rows, so take per-modality numbers from table 4 [arxiv:2112.04482 tab. 4]. On ImageNet linear evaluation the released CLIP model is ahead, 80.2 against 75.5 [arxiv:2112.04482 tab. 5], so the masked objectives do not win on vision alone. Card: [FLAVA: A Foundational Language And Vision Alignment Model](/papers/2112.04482). ### Masked regression to teacher features data2vec uses one recipe for speech, images and text. An EMA teacher sees the unmasked input and averages the outputs of its top K = 10 blocks, with each block normalised [arxiv:2202.03555 §4.3] [arxiv:2202.03555 §3.3]. A student sees the masked input and regresses those averages at the masked steps with a smooth L1 loss, beta 4 [arxiv:2202.03555 §3.3] [arxiv:2202.03555 §4.3]. The target is the teacher's representation, not the token, so no tokenizer is needed for the target [arxiv:2202.03555 §3.3]. The paper's main text setup is BERT masking, 15% with the 80/10/10 rule [arxiv:2202.03555 §4.3]. The released text config uses spans of 4 at probability 0.35, with no random or unmasked tokens [code: examples/data2vec/config/text/pretraining/base.yaml#L25-L26]. On GLUE dev, the single-task average is 82.7 for data2vec and 82.5 for a RoBERTa baseline retrained in the same setup [arxiv:2202.03555 tab. 4]. Card: [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](/papers/2202.03555). ### Cloze prompts for a fixed masked model This is an inference-side use. The masked model is frozen. A relation such as date of death is queried with a cloze prompt that has a blank for the subject and a blank for the object [arxiv:2104.06599 §3]. The paper replaces the prompt tokens with continuous vectors, adds a trainable perturbation at every layer, and fits a mixture of such prompts with EM for each relation [arxiv:2104.06599 §3.1–§3.5]. On T-REx with BERT-large, the mixture reaches P@1 51.6, against 39.4 for the LPAQA baseline, which the authors reimplemented [arxiv:2104.06599 tab. 1]. Card: [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](/papers/2104.06599). ### Masked diffusion over text Masking-only diffusion treats masking as the noise process. Each token is replaced by the mask token independently, with a probability set by a schedule, and a masked token stays masked [arxiv:2406.07524 §3.1] [arxiv:2406.07524 §3.2.1]. The denoiser is a diffusion transformer with timestep conditioning [arxiv:2406.07524 §3.5.1], and its training input is the masked sequence, so the step is close to MLM. The paper's continuous-time loss is a weighted average of masked-token losses [arxiv:2406.07524 §3.5]. In the experiments the masking probability at time t is t, which gives a weight of 1/t [arxiv:2406.07524 app. D.2]. That weight is derived from the paper's eq. 10 and is not written out there [arxiv:2406.07524 §3.5]. Generation starts from a row of mask tokens and unmasks over N steps. Each masked token is revealed with a probability set by the schedule, and the unmasking is random rather than ranked by confidence [arxiv:2406.07524 §4.1]. The code's default is 128 sampling steps, and its OWT generative-perplexity script uses 1,000 [code: configs/config.yaml#L30-L31] [code: scripts/train_owt_mdlm.sh#L27]. Longer text uses a block-by-block variant with a clean prefix [arxiv:2406.07524 §4.2]. Card: [Simple and Effective Masked Diffusion Language Models](/papers/2406.07524). The reported likelihoods are upper bounds. On LM1B the 327B-token bound is 23.00, against 20.86 for a retrained autoregressive model [arxiv:2406.07524 tab. 1]. On OpenWebText the bound is 23.21, against 17.54 for autoregressive [arxiv:2406.07524 tab. 2]. Bounds from diffusion models are not directly comparable with exact autoregressive likelihoods [arxiv:2406.07524 tab. 1]. The semi-autoregressive sampler reaches a generative perplexity of 27.18 at 89.3 seconds per sequence, against 35.43 at 2473.9 seconds for SSD-LM, on 200 sequences of 2,048 tokens on one GPU [arxiv:2406.07524 tab. 5]. ### Iterative masked decoding over discrete tokens MoMask uses masking for the base layer of a residual quantizer. A bidirectional transformer over base tokens is conditioned on CLIP text features. The mask ratio follows a cosine schedule, and the loss is the negative log-likelihood of the original tokens at masked positions [arxiv:2312.00063 §3.2] [arxiv:2312.00063 eq. 3]. At inference, all base tokens start masked. The model fills them in parallel over L iterations, and the least confident fills are masked again on each round. The paper uses L = 10 [arxiv:2312.00063 §3.4] [arxiv:2312.00063 §4]. A second bidirectional transformer then predicts the finer residual layers one at a time [arxiv:2312.00063 §3.4]. The introduction says all layers are produced in about 15 iterations [arxiv:2312.00063 §1]. On HumanML3D, the paper reports FID 0.045 for MoMask against 0.141 for T2M-GPT [arxiv:2312.00063 tab. 1]. The ablation table lists 0.051 for the same setting [arxiv:2312.00063 tab. 2]. The authors report that generation quality starts to degrade above five residual layers [arxiv:2312.00063 §4.2]. The same inference procedure, with a time range masked, fills in a gap inside an existing motion [arxiv:2312.00063 §4.3]. Card: [MoMask: Generative Masked Modeling of 3D Human Motions](/papers/2312.00063). ### Self-critical tuning of a masked headline generator The ad-headline model is a BERT-Large model over a set of product titles. Its hidden positions are only in the headline, and the source titles are never masked in the main model [arxiv:2607.06818 §3.2]. Stage 1 trains with the masked-token cross-entropy over those positions [arxiv:2607.06818 §3.1] [arxiv:2607.06818 eq. 2]. The masking rate is not stated [arxiv:2607.06818 §3.1]. Stage 2 is self-critical. For each campaign, one headline is sampled and one is decoded greedily. The sampled headline is pushed up only when its ROUGE-L F1 against the approved headline beats the greedy one [arxiv:2607.06818 §3.3] [arxiv:2607.06818 eq. 9] [arxiv:2607.06818 §4.1]. Inference uses beam search with a length-normalised score [arxiv:2607.06818 §3.4] [arxiv:2607.06818 eq. 10]. The paper reports ROUGE-L 6.33 above a bi-LSTM baseline for the full model, and 5.08 for the MLM stage alone [arxiv:2607.06818 tab. 1]. Its table values are differences from that baseline. The reward is the same metric as the top overlap row, so part of that gain reflects the metric being optimised. The crowd audit is the independent check: 6.53% more headlines are rated 3 of 3 than human-submitted ones [arxiv:2607.06818 tab. 2]. The data is proprietary, English only [arxiv:2607.06818 §4.1], and the card records no official code. Card: [Ad Headline Generation using Self-Critical Masked Language Model](/papers/2607.06818). ### Mask-filling probe of an encoder To ask whether a masked encoder leans politically, the probe places each political statement inside a fixed request with one mask for the stance word. The model returns its ten most likely fill-ins. The mass on a positive lexicon is compared with the mass on a negative one, and the answer is "strong" if the gap is at least 0.3 [arxiv:2305.08283 §2.1] [arxiv:2305.08283 app. A.1]. The lexicons were built by hand from 100 example outputs, and the 0.3 threshold was set by hand [arxiv:2305.08283 app. A.1]. The repository has no encoder code for this probe [code: step1_response.py]. The same paper then continues pretraining RoBERTa on partisan corpora and fine-tunes it on hate-speech and misinformation tasks [arxiv:2305.08283 §4.1] [arxiv:2305.08283 §4.2]. Overall gaps are small. On Hate-Demographic, balanced accuracy is 90.26 for vanilla RoBERTa and 89.30 for the right-leaning news variant [arxiv:2305.08283 tab. 3]. The per-group differences are the paper's main result [arxiv:2305.08283 §4.2] [arxiv:2305.08283 tab. 4]. Card: [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](/papers/2305.08283). ### Masked-entity bias benchmark CAMeL replaces a culture-specific entity with a mask token in each of its prompts [arxiv:2305.14456 §3.2] [arxiv:2305.14456 tab. 1]. It has 628 Arabic prompts, 250 of them contextualised to Arab culture [arxiv:2305.14456 §3]. For BERT-type models the full prompt is scored with its mask token. For GPT- and T5-type models only the text before the mask is used [arxiv:2305.14456 §4.4]. The Cultural Bias Score is the share of Arab–Western entity pairs where the model rates the Western entity higher, and zero is ideal [arxiv:2305.14456 §4.4]. Author-reported, on the contextualised prompts, ARBERT scores 46.40 and AraBERT-T (base) scores 50.27 [arxiv:2305.14456 tab. 10]. The paper rewrote some prompts for GPT-type models so the Arab cue comes before the mask, so their scores use different prompts from the masked-LM scores [arxiv:2305.14456 app. C.1]. The prompts derive from Twitter/X posts, and the paper says they are masked before release, so check the data's terms before reuse [arxiv:2305.14456 Ethics Statement]. Card: [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](/papers/2305.14456). ### Stereotype fill-in scoring StereoSet's intrasentence test has a blank in a context. Each of its three options is a stereotype, an anti-stereotype or an unrelated option [arxiv:2004.09456 §2] [arxiv:2004.09456 §2.1]. For BERT, RoBERTa and XLNet, each attribute subword is scored with its own masked prompt, with earlier subwords visible [code: dataloader.py#L73-L82]. The card's paper-vs-code notes say the paper averages log probabilities, while the code averages raw probabilities [arxiv:2004.09456 §7.1] [code: code/eval_discriminative_models.py#L176-L177]. Both sub-scores only rank options within each item [arxiv:2004.09456 §6.2]. ICAT combines a language-model score and a stereotype score [arxiv:2004.09456 §6.2]. The author-reported dev-split ICAT is 69.4 for BERT-base and 71.9 for GPT-2 small [arxiv:2004.09456 tab. 4]. The test split is hidden [arxiv:2004.09456 §6.1], and the repository has no test CATs [code: data/]. Card: [StereoSet: Measuring stereotypical bias in pretrained language models](/papers/2004.09456). ### Multilingual evaluation with masked encoders Belebele is a test set, not a training method. Its 900 four-option reading items are parallel across 122 language variants [arxiv:2308.16884 §3.5]. Masked encoders are evaluated in two ways: fine-tuned on English, or fine-tuned on machine-translated training data for every target language [arxiv:2308.16884 §4.2] [arxiv:2308.16884 app. A.4.1]. For XLM-V, the translate-train setting averages 60.2 across languages, against 55.6 for English-only fine-tuning [arxiv:2308.16884 tab. 2]. LLaMA 2 70B with five English examples averages 48.0 over the same languages, although it reaches 90.9 on English [arxiv:2308.16884 tab. 2]. The benchmark is CC BY-SA, but its English training set is assembled from CC BY-NC sources, so commercial fine-tuning on it is not permitted [arxiv:2308.16884 app. A.3]. Use the test set only, and do not reuse the training set for commercial models [arxiv:2308.16884 §3.4]. Card: [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](/papers/2308.16884). ### Pseudo-likelihood scoring of masked models The BabyLM shared task scores grammar with zero-shot minimal pairs. The model is correct when it gives the acceptable sentence the higher score [arxiv:2504.08165 §5.1]. Masked models are scored by pseudo-likelihood, following Salazar et al. (2020), through the minicons implementation [arxiv:2504.08165 §5.2.1]. The paper's organisers scored 31 teams' submissions on the same pipeline, and the reported scores are author-reported [arxiv:2504.08165 §5.3]. The card does not state whether the baseline models were trained with MLM, so this variant is evaluation only [arxiv:2504.08165 §5.4]. Card: [Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](/papers/2504.08165). ### Masked-LM fillers for black-box adversarial text BAE attacks a text classifier in the black-box, soft-label setting. The attacker may query output probabilities, but not parameters, gradients or training data [arxiv:2004.01970 §2]. Each token is scored by how much deleting it lowers the probability of the true label, following TextFooler and Ren et al. [arxiv:2004.01970 §2]. Tokens are visited in that order. Each is masked, and the top 50 predictions of a pretrained BERT base-uncased masked LM become candidates [arxiv:2004.01970 §2] [arxiv:2004.01970 app. A]. Candidates pass a Universal Sentence Encoder similarity filter with a threshold of 0.8, and replacements also pass a part-of-speech check [arxiv:2004.01970 app. A]. The search stops when the label flips [arxiv:2004.01970 alg. 1]. The four modes are BAE-R (replace), BAE-I (insert left or right of a token), BAE-R/I (choose one per token) and BAE-R+I (replace, then insert at the same position) [arxiv:2004.01970 §2]. The paper does not state how the insert side is chosen, or how BAE-R/I chooses between replace and insert [arxiv:2004.01970 §2]. Author-reported results cover seven datasets and three targets: word-LSTM, word-CNN and BERT base-uncased [arxiv:2004.01970 §3] [arxiv:2004.01970 tab. 1] [arxiv:2004.01970 tab. 2]. BAE-R+I gives the lowest accuracy in most settings. BAE-I has the highest average similarity among the four modes [arxiv:2004.01970 §3]. For BERT as the target, TextFooler gives lower accuracy than BAE-R on six of seven datasets. BAE-R is the stronger attack only on TREC [arxiv:2004.01970 tab. 1] [arxiv:2004.01970 tab. 2]. In a human study on three datasets with the BERT target, raters scored BAE-R examples higher on naturalness than TextFooler examples, 3.91 against 3.17 on Amazon, on a 1 to 5 scale [arxiv:2004.01970 tab. 3] [arxiv:2004.01970 app. C]. The early-stopping step uses the test split, so reported accuracy may be optimistic [arxiv:2004.01970 app. A]. Amazon, Yelp and IMDB each have 100 test points, so one point moves accuracy by about one percentage point [arxiv:2004.01970 tab. 5]. Card: [BAE: BERT-based Adversarial Examples for Text Classification](/papers/2004.01970). ### Blank infilling at 130B scale GLM-130B is a bidirectional model trained by blank infilling [arxiv:2210.02414 §2.1]. In 30% of training sequences, several short spans are marked with [MASK]. Span lengths are drawn from Poisson(3), and about 15% of tokens are hidden [arxiv:2210.02414 §2.2]. In the other 70%, a prefix is kept, and the tail is hidden behind one [gMASK] span with a uniformly drawn length [arxiv:2210.02414 §2.2]. The spans are written back in a random order, and each one sees the spans already written [arxiv:2210.02414 §2.1]. The paper's training-stability fixes are DeepNorm Post-LN, an embedding gradient shrink of 0.1, and FP16 training [arxiv:2210.02414 §3]. Author-reported, zero-shot LAMBADA accuracy is 80.21, against 76.2 for GPT-3 175B [arxiv:2210.02414 §5.1]. Weight-only INT4 quantization gives 79.47 on LAMBADA and 44.80 on MMLU, against 80.21 and 44.75 in FP16 [arxiv:2210.02414 §4]. The released weights carry a non-commercial model license, separate from the Apache-2.0 code, so check the card's Pitfalls before reuse. Card: [GLM-130B: An Open Bilingual Pre-trained Model](/papers/2210.02414). ### Masked-LM pretraining with linear-cost attention FLASH replaces the Transformer block with a gated attention unit, and splits attention into exact attention inside fixed-size chunks and linear attention between chunks [arxiv:2202.10447 §2] [arxiv:2202.10447 §3.2]. Its C4 MLM setting is one of the training regimes in the paper [arxiv:2202.10447 tab. 6] [arxiv:2202.10447 tab. 7]. At context 8192, the reported per-step latency on 64 TPU-v4 cores is 59.9 ms for FLASH and 252.1 ms for Transformer++, with C4 MLM perplexity 3.897 against 3.933 [arxiv:2202.10447 tab. 12]. The training-cost speed-up over Transformer++ is 1.0x to 4.8x across contexts 512 to 8192 [arxiv:2202.10447 §4.1]. These are wall-clock figures, not per-step speed-ups, and the card does not give the masking rate or loss for the C4 MLM runs. The card records no official code, so those details cannot be checked against one [arxiv:2202.10447 tab. 6]. Card: [Transformer Quality in Linear Time](/papers/2202.10447). ### Masked SMILES pretraining against descriptor regression ChemBERTa-2 is a RoBERTa encoder over canonical SMILES strings, with a 591-token vocabulary and inputs of up to 512 tokens [arxiv:2209.01712 §3.1]. Its MLM step masks 15% of the tokens and predicts the originals [arxiv:2209.01712 §3.1]. The rival objective, multi-task regression (MTR), needs no masking: it predicts 200 RDKit properties computed from each string [arxiv:2209.01712 §3.1]. Both are run on 5M, 10M and 77M compounds [arxiv:2209.01712 §3]. The authors' Table 1 compares the two at 77M. MTR-77M is ahead of MLM-77M on all eight columns, for example BACE RMSE 1.363 against 1.509 and Lipophilicity RMSE 0.798 against 0.987 [arxiv:2209.01712 tab. 1]. The MLM rows also have ClinTox ROC-AUC values of 0.239 to 0.349, below chance, which the paper does not discuss [arxiv:2209.01712 tab. 1]. The authors conclude that MTR tends to fine-tune better [arxiv:2209.01712 §4.1]. The paper does not state the MLM replacement rule; the training code takes it from the HuggingFace collator [code: chemberta/train/utils.py#L37-L47]. The paper gives no GPU-hour total [arxiv:2209.01712 §7.1]. Card: [ChemBERTa-2: Towards Chemical Foundation Models](/papers/2209.01712). ### Weak links Three cards use masking or masked models without an MLM training objective. They are listed here so that a reader does not take them as MLM evidence. - **Full-sequence diffusion as partial masking.** Diffusion Forcing reads full-sequence diffusion as partial masking of every token, and gives each token its own mask level [arxiv:2407.01392 §3.1]. It trains a causal recurrent denoiser, not a masked encoder, so it is a framing only. Card: [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](/papers/2407.01392). - **Pretrained chemical language models.** BOOM evaluates ChemBERTa, MoLFormer and Regression Transformer on out-of-distribution molecules. Pretraining improved in-distribution RMSE by 31%, 35% and 12%, with no significant out-of-distribution gain [arxiv:2505.01912 §4.2]. The card does not state these models' pretraining objectives. Card: [BOOM](/papers/2505.01912). - **Fine-tuned MARBERT on Arabic tweets.** MARBERT is the masked encoder here, but the method is a classifier fine-tuned from it. No masked-token step is described, so the card is evidence of downstream use, not of MLM training [arxiv:2606.25495 §4.2] [arxiv:2606.25495 §5.2]. The labelled spam set uses labels from a keyword list, so its high score may partly reflect that list [arxiv:2606.25495 §5.1]. Card: [Spam and Sentiment Detection in Arabic Tweets Using MARBERT Model](/papers/2606.25495). ## When to use which | Situation | Use | Why | |---|---|---| | Pretraining an encoder on text whose word boundaries you can recover from the raw text | Whole-word masking | Small gain in the authors' ablation: +0.5 XNLI, near-equal POS and parsing, NER slightly lower (91.44 vs 91.55) [arxiv:1911.03894 tab. 8] [arxiv:1911.03894 app. A]. Single pretraining seed | | Text, but no reliable word boundaries, or you cannot keep the word-to-subword mapping through tokenisation | Subword masking | The baseline needs no mapping [arxiv:1911.03894 §4.3] [arxiv:1911.03894 tab. 8]. FlauBERT used this unit at 71 GB scale [arxiv:1912.05372 tab. 1] | | Sequences of discrete codes, such as visit diagnoses, where you pretrain on histories before a prediction task | Masked code prediction with the BERT 15% and 80/10/10 rule | The paper pretrains this way and then fine-tunes for next-visit prediction [arxiv:1907.09538 §2.4] [arxiv:1907.09538 §2.5]. Reported gains over DeepR and RETAIN are on one UK primary-care dataset [arxiv:1907.09538 tab. 1] | | Multi-label classifier where some labels may be known at inference | Label masking | The model accepts known label states and predicts the rest in one pass [arxiv:2011.14027 §4.2]. The paper also reports results with partial labels at test time [arxiv:2011.14027 tab. 3] | | Fixed masked model, labelled pairs for one relation, and gradient access to the model | Cloze prompts with soft prompts | The method trains prompt vectors by gradient descent [arxiv:2104.06599 §3]. Reported P@1 gain over the LPAQA baseline on T-REx [arxiv:2104.06599 tab. 1] | | Limited pretraining text, a few GB of web text | Web text can be enough, and the corpus choice matters more than its size | A 4GB OSCAR sample is close to the 138GB corpus on fine-tuned POS, parsing and XNLI, but 138GB is +0.9 F1 on NER [arxiv:1911.03894 tab. 5] [arxiv:1911.03894 §6.2] | | Pretraining a French encoder at scale and comparing against a multilingual model | Subword MLM on a large French corpus, with no next-sentence prediction | Author-reported: both FlauBERT sizes beat mBERT on most FLUE tasks [arxiv:1912.05372 §5] [arxiv:1912.05372 tab. 3]. The masking rate is not stated [arxiv:1912.05372 tab. 1] | | Discrete tokens from a trained quantizer, text-conditioned generation, few sampling steps | Iterative masked decoding, base layer first, then residual layers | Ten base iterations [arxiv:2312.00063 §4]. Author-reported FID 0.045 against 0.141 for T2M-GPT on HumanML3D [arxiv:2312.00063 tab. 1]. Needs a trained residual VQ tokenizer first [arxiv:2312.00063 §3.1] | | Text generation that must be non-autoregressive, and sampling cost is acceptable | Masked diffusion (MDLM) | Likelihoods are upper bounds that trail autoregressive baselines [arxiv:2406.07524 tab. 1] [arxiv:2406.07524 tab. 2]. Semi-autoregressive sampling is faster than SSD-LM in the authors' setup [arxiv:2406.07524 tab. 5] | | You need exact likelihoods, or the lowest perplexity on text | Autoregressive model | The masked diffusion bounds are higher than the retrained autoregressive numbers on both LM1B and OWT [arxiv:2406.07524 tab. 1] [arxiv:2406.07524 tab. 2] | | Checking whether an encoder's mask-fill answers carry a political lean | Mask-filling probe with a stance lexicon | The probe is described in [arxiv:2305.08283 §2.1], but its lexicons and threshold were set by hand and the repository has no encoder code. Changing the threshold changes every label [arxiv:2305.08283 app. A.1] | | Multilingual encoder evaluation across many languages | Belebele, test set only, with a multiple-choice head | 900 items per language, parallel across 122 variants [arxiv:2308.16884 §3.5]. Translate-train averages are the strongest encoder setting [arxiv:2308.16884 tab. 2]. The training set is non-commercial, so do not use it for commercial models [arxiv:2308.16884 app. A.3] | | Unlabelled sentences only, and you want sentence embeddings without pairs | Sentence reconstruction by deletion (TSDAE), not MLM | Author-reported unsupervised average 55.2 against 52.9 for MLM [arxiv:2104.06979 tab. 2]. Training uses a decoder, but only the encoder is used at inference [arxiv:2104.06979 §3.2] | | Textual graph where each node's text should reflect its neighbourhood, and you can afford long training | Masked span warm-up, then fine-tuning on clean inputs | Removing the progressive stage costs 0.98 to 1.71 P@1 points [arxiv:2105.02605 §4.4]. Reported training takes 58.8 h on Product to 151.2 h on Wiki on eight V100s [arxiv:2105.02605 app. B] | | Text linked to a knowledge graph, and you can pretrain for about a week on eight A100s | Joint MLM and link prediction | MLM alone scores 74.3 on CSQA against 76.0 for the joint objective [arxiv:2210.09338 tab. 5]. The authors' gains are over a continued-MLM RoBERTa baseline, not the released checkpoint [arxiv:2210.09338 §3.3] | | Documents with real links between them (hyperlinks, citations), and days of pretraining on A100s | Two-segment inputs with a relation head (LinkBERT) | Author-reported MRQA average 77.8 against 75.2 for BERT-base [arxiv:2203.15827 tab. 1]. Dropping the relation loss lowers base scores on the datasets reported [arxiv:2203.15827 tab. 6]. Graph-building cost is not stated [arxiv:2203.15827 §6.1] | | Parallel sentence pairs for a multilingual encoder | Translation language modeling | The survey reports +4.9% machine-translation gain for TLM, against +1.3% for MLM alone without parallel text [arxiv:2103.11943 p. 10]. Figures are second-hand | | Many passes over the same text, and you want a fresh mask on each pass | Dynamic masking | Part of the RoBERTa recipe, whose combined gain over BERT is +3 to +4 points. The masking rate is not stated [arxiv:2103.11943 p. 11–12] | | Measuring cultural bias of masked models on entity slots | Masked-entity CBS on CAMeL | Author-reported CBS of 46.40 for ARBERT on the contextualised prompts [arxiv:2305.14456 tab. 10]. Do not compare GPT-type scores with masked-LM scores, because the prompts differ [arxiv:2305.14456 app. C.1] | | Zero-shot grammaticality ranking with a masked model | Pseudo-likelihood scoring | Follows Salazar et al. through minicons [arxiv:2504.08165 §5.2.1]. The paper's pipeline scores are author-reported [arxiv:2504.08165 §5.3] | | Single-language encoder for German, with a large German corpus and about 1.2 days of pod time for base | Monolingual RoBERTa-style MLM; masking rate not stated | Base models are best on four of six German tasks [arxiv:2012.02110 tab. 3]. Filtering made no clear difference [arxiv:2012.02110 tab. 3]. The masking rate and unit are not stated [arxiv:2012.02110 §3] | | Single-language encoder for Spanish, compared with a multilingual model | Whole-word and dynamic MLM, cased or uncased by task | Author-reported: BETO beats the best mBERT on most tasks, but not on every QA setting [arxiv:2308.02976 tab. 1] [arxiv:2308.02976 tab. 2]. Cased is best on XNLI, NER and POS, uncased on PAWS-X and MLDoc [arxiv:2308.02976 tab. 1]. Masking rate not stated [arxiv:2308.02976 §3] | | Categorical tabular rows with few labels, and you can pretrain on unlabeled rows | Masked feature prediction (MLM) on larger tables, replaced-value detection on smaller ones | On the 8 larger datasets, MLM is ahead of the best other method at 50, 200 and 500 labels [arxiv:2012.06678 tab. 3]. On the 12 smaller datasets, the replaced-value variant is ahead at every label count [arxiv:2012.06678 tab. 4] | | Noisy image-caption pairs, with a fusion encoder and enough memory for a second copy of the weights | MLM with momentum distillation | Adds 0.36 VQA test-dev points and 0.10 NLVR² test-P points over the same setup with plain MLM [arxiv:2107.07651 tab. 1]. The EMA copy doubles weight memory [code: models/model_pretrain.py#L63-L76] | | Several modalities under one training recipe, and you accept that its text gain is small | Regression to teacher features (data2vec) | On GLUE dev the average is 82.7 against 82.5 for a retrained RoBERTa baseline [arxiv:2202.03555 tab. 4]. Text masking in the paper's main results is 15% with 80/10/10 [arxiv:2202.03555 §4.3] | | Measuring stereotypes in an English model on natural sentences, and you can use only the public dev split | Stereotype fill-in scoring with StereoSet | The dev split is public and the test split is hidden [arxiv:2004.09456 §6.1]. The masked-LM score differs between paper and code [arxiv:2004.09456 §7.1] [code: code/eval_discriminative_models.py#L176-L177] | | Question answering over tables, with a parser that can take a learned encoder, and compute for pretraining on tables | TaBERT-style encoder: MLM plus masked column prediction and cell value recovery, with K = 3 rows | Author-reported WikiTableQuestions test 52.3% for the best dev run, against 49.2% for MAPO with BERT-Base at K = 1 [arxiv:2005.08314 tab. 1]. On Spider dev, TranX with TaBERT-Large at K = 3 scores 64.5 against 61.3 with BERT-Large at K = 1 [arxiv:2005.08314 tab. 2]. The largest model took about six days on 128 V100 GPUs [arxiv:2005.08314 app. A.2] | | Black-box attack on a text classifier that returns probabilities, where substitutions must read naturally | BAE-R or BAE-R+I, with BERT base-uncased fillers | BAE-R+I is the strongest mode in most settings [arxiv:2004.01970 §3] [arxiv:2004.01970 tab. 1]. Human raters scored BAE-R more natural than TextFooler on three datasets [arxiv:2004.01970 tab. 3]. For BERT targets, TextFooler is the stronger attack on six of seven datasets with replacement only [arxiv:2004.01970 tab. 1] [arxiv:2004.01970 tab. 2] | | Arabic encoder for formal text, or for dialect and social-media text where tweets matter | ARBERT for formal text, MARBERT for tweets | Author-reported ARLUE: MARBERT-v2 scores 77.40 against 76.55 for XLM-R Large [arxiv:2101.01785 §5.2]. The text never defines MARBERT-v2, and the checkpoints are for research use only [arxiv:2101.01785 §3.2] | | Encoder pretraining where every token should give a training signal, and you can afford a generator and a discriminator | Replaced-token detection with gradient-disentangled sharing (DeBERTaV3) | Dev-set GLUE average 91.37 for DeBERTaV3-Large against 90.00 for DeBERTa-large [arxiv:2111.09543 tab. 3]. Set λ explicitly: the paper uses 50, and the released code defaults to 10 [arxiv:2111.09543 §3.1] [code: DeBERTa/apps/tasks/rtd_task.py#L392] | | Classifying Arabic tweets for sentiment or spam, with labelled tweets of your own | Fine-tune MARBERT with a classification head; no masked-token step | Author-reported sentiment F1 0.75 at batch 96, learning rate 2e-5 and 5 epochs [arxiv:2606.25495 tab. 4]. Sarcasm and indeterminate stay weak, at F1 0.54 and 0.56 [arxiv:2606.25495 tab. 5]. The paper runs no baseline on its own data [arxiv:2606.25495 tab. 2] [arxiv:2606.25495 tab. 3] | | Dutch encoder for NER, tagging, semantic roles or sentiment, with a monolingual corpus of a few GB | Monolingual BERT-base with sentence order prediction (BERTje) | Author-reported CoNLL-2002 test F1 88.3 against 80.7 for multilingual BERT [arxiv:1912.09582 tab. 1]. Single runs. Whole-word masking in the paper is off in the code, and the code caps predictions at 20 per sequence [code: pretraining/create-pretrain-data.py#L34] | | Molecules as SMILES strings, scarce labels, and pretraining compute for tens of millions of strings | Descriptor regression (MTR) over MLM, if RDKit can compute the targets | At 77M, MTR is ahead of MLM on all eight Table 1 columns [arxiv:2209.01712 tab. 1]. The 77M set is not in the repository, and the five configurations kept for 10M and 77M are not listed [arxiv:2209.01712 §3.2] [code: chemberta/data/] | | Image-text model that must also work on text or on images alone, trained from public data only | Masked multimodal plus unimodal MLM (FLAVA) | Author-reported macro average 75.85 against 66.78 for CLIP ViT-B/16 trained on 400M pairs [arxiv:2112.04482 tab. 3]. The released CLIP is ahead on ImageNet linear evaluation, 80.2 against 75.5 [arxiv:2112.04482 tab. 5] | | Bidirectional English and Chinese generative model, with a large compute budget, and a model licence that allows your use | Blank infilling with [MASK] and [gMASK] (GLM-130B) | Author-reported LAMBADA 80.21 against 76.2 for GPT-3 175B, and 79.47 with INT4 weights [arxiv:2210.02414 §5.1] [arxiv:2210.02414 §4]. The card's Pitfalls note the non-commercial weight licence | | Short generated text with a scoring metric you can compute on approved examples, and a pretrained masked encoder to start from | Masked-LM pretraining, then self-critical tuning with the metric as reward | Author-reported ROUGE-L 6.33 above a bi-LSTM baseline, and 5.08 for the MLM stage alone [arxiv:2607.06818 tab. 1]. The reward is the reported metric, so check gains on the crowd audit [arxiv:2607.06818 tab. 2]. One 30-epoch run took over 20 days on a V100 [arxiv:2607.06818 §4.1] | | Long inputs where attention cost dominates, and you can run a non-standard attention kernel | Chunked linear-cost attention (FLASH) with MLM or LM | Author-reported training-cost speed-up 1.0x to 4.8x on C4 MLM [arxiv:2202.10447 §4.1]. Per-step latency at 8192 is 59.9 ms against 252.1 ms for Transformer++ [arxiv:2202.10447 tab. 12]. No official code, and the card does not state the C4 MLM masking rate | The CamemBERT ablation runs use one pretraining seed per model, and the authors attribute some of the differences to seed noise [arxiv:1911.03894 §6.2]. Treat differences of a few tenths of a point as unresolved. ## Implementation sketch Five shapes appear in the cards. The first covers token and word masking [arxiv:1911.03894 §4.3]. Masked diagnosis codes follow the same rule, with a code in place of a token [arxiv:1907.09538 §2.4]. ```pseudocode # Select units to hide, one sequence [arxiv:1911.03894 §4.3] # units: groups of positions. One position each for subword masking; # all subword pieces of one word for whole-word masking def mask_sequence(tokens, units): targets = {} for unit in units: # Select about 15% of units [arxiv:1911.03894 §4.3] if random_value() >= 0.15: continue # The loss is taken at every position of a selected unit [arxiv:1911.03894 §4.3] for index in unit: targets[index] = tokens[index] # Draw once per unit: this is a choice of ours, see the notes below draw = random_value() for index in unit: if draw < 0.8: tokens[index] = MASK_TOKEN # 80% [arxiv:1911.03894 §4.3] elif draw < 0.9: tokens[index] = random_token() # 10% [arxiv:1911.03894 §4.3] # else: keep the original token # 10% [arxiv:1911.03894 §4.3] return tokens, targets ``` The second covers label masking in a multi-label classifier [arxiv:2011.14027 §3.4]. ```pseudocode # Hide a random share of the L labels for one training image [arxiv:2011.14027 §3.4] def label_mask_step(image, labels): # labels: (L,) with 0 negative, 1 positive # Draw the number of hidden labels between 0.25L and L [arxiv:2011.14027 §3.4] n_hidden = draw_count(low=0.25 * L, high=L) hidden = random_subset(L, n_hidden) # Hidden labels get the unknown state; the rest keep their ground truth [arxiv:2011.14027 §3.4] state = labels.copy() state[hidden] = UNKNOWN logits = forward(image, state) # one pass over all label tokens [arxiv:2011.14027 §3.2] # Binary cross-entropy only on the hidden labels [arxiv:2011.14027 §3.4] return bce(logits[hidden], labels[hidden]).mean() ``` The third covers iterative decoding for discrete tokens, following the MoMask base layer [arxiv:2312.00063 §3.4]. Masked diffusion differs in one rule: it unmasks at random with the schedule's probability instead of re-masking the least confident tokens [arxiv:2406.07524 §4.1]. ```pseudocode # Iterative masked decoding of the base layer [arxiv:2312.00063 §3.4] # The paper runs L = 10 iterations [arxiv:2312.00063 §4] def generate_base_tokens(text, length, steps): # Start with every base-layer position masked [arxiv:2312.00063 §3.4] tokens = full(length, MASK_ID) # (n,) for step in range(steps): # Predict all masked positions at once [arxiv:2312.00063 §3.4] logits = masked_transformer_cfg(tokens, text) # (n, K) sampled = sample_categorical(logits) # (n,) # Keep the new samples only at masked positions [arxiv:2312.00063 §3.4] is_masked = tokens == MASK_ID tokens = where(is_masked, sampled, tokens) # Confidence of each new sample; fixed tokens are never re-masked [arxiv:2312.00063 §3.4] confidence = where(is_masked, prob_of(logits, sampled), INFINITY) # Re-mask the least confident tokens, cosine schedule [arxiv:2312.00063 §3.4] count = ceil(cos(pi * (step + 1) / (2 * steps)) * length) lowest = argsort(confidence)[:count] tokens[lowest] = MASK_ID return tokens ``` The fourth is the masked-diffusion denoising loss on text [arxiv:2406.07524 §3.5] [arxiv:2406.07524 app. D.2]. Its weight is 1/t under the log-linear schedule, as noted above. ```pseudocode # One masked-diffusion training step on text [arxiv:2406.07524 alg. 1] # Input: tokens (B, L) of token ids; pad_mask (B, L), 1 for real tokens def mdlm_training_step(tokens, pad_mask): # Draw one masking time per sequence [arxiv:2406.07524 app. D.3] times = uniform(B, low=eps, high=1.0) # (B,) # Mask each token independently with probability t [arxiv:2406.07524 app. D.2] is_masked = uniform(B, L) < times[:, None] # (B, L) noisy = where(is_masked, MASK_ID, tokens) # (B, L) # Log-probability of the clean token; 0 at unmasked [arxiv:2406.07524 §3.2.3] log_probs = subs(backbone(noisy, times), noisy) # (B, L, V) true_logp = gather(log_probs, tokens) # (B, L) # Weight 1/t from eq. 10 under the log-linear schedule [arxiv:2406.07524 §3.5] per_token = -(1.0 / times[:, None]) * true_logp # (B, L) return sum(per_token * pad_mask) / sum(pad_mask) # token average ``` The fifth covers two-segment pretraining with a relation head, as in LinkBERT. It adds one term to the first sketch [arxiv:2203.15827 §4.2] [arxiv:2203.15827 eq. 1]. ```pseudocode # MLM plus a relation loss on [CLS], as in LinkBERT [arxiv:2203.15827 §4.2] # Input: tokens (B, n) for [CLS] X_A [SEP] X_B [SEP]; relations (B,) in {0,1,2} def linkbert_loss(model, tokens, relations): # Mask 15% of tokens: 80% [MASK], 10% random, 10% kept [arxiv:2203.15827 §3] masked, targets = apply_mlm_masking(tokens) hidden = model.encode(masked) # (B, n, d) mlm = masked_cross_entropy(model.mlm_head(hidden), targets) # Three-way relation head on [CLS], replacing NSP [arxiv:2203.15827 §4.2] drp = cross_entropy(model.drp_head(hidden[:, 0]), relations) # (B, 3) # Equal weights, 1:1 [arxiv:2203.15827 eq. 1] return mlm + drp ``` Where the papers leave this open: - **Dynamic or static masks.** CamemBERT redraws masks each time a sequence is used [arxiv:1911.03894 §4.3]. FlauBERT also uses dynamic masking [arxiv:1912.05372 tab. 1]. The first sketch above does not say which to use. - **Draw per unit or per token.** For whole-word masking, the 80/10/10 draw is not stated as per word or per subword piece [arxiv:1911.03894 §4.3]. The sketch draws per unit, which is our choice. - **Random replacement.** CamemBERT says "a random token" without saying how it is drawn [arxiv:1911.03894 §4.3]. - **Masking rate in FlauBERT.** The text does not state it [arxiv:1912.05372 tab. 1]. BEHRT and CamemBERT both use 15% [arxiv:1907.09538 §2.4] [arxiv:1911.03894 §4.3]. - **Distribution of the hidden-label count.** The card gives the range for label masking but does not restate the distribution [arxiv:2011.14027 §3.4]. - **Mask-ratio rounding and replacement split in MoMask.** The paper writes a ceiling for the mask count and an 80/10/10 split [arxiv:2312.00063 §3.2]. The released code rounds the count with a minimum of 1, and applies 10% random tokens and then 88% of the rest as masks, which is about 79/10/11 [arxiv:2312.00063 §3.2] [code: models/mask_transformer/transformer.py#L291-L299]. - **Re-mask schedule in MoMask.** The paper indexes the schedule by iteration count [arxiv:2312.00063 §3.4]. The released code maps steps onto an evenly spaced grid from 0 to 1, so the step-to-ratio mapping differs [code: models/mask_transformer/transformer.py#L363-L365]. The sketch uses the paper's text. - **Noise type in TSDAE.** The text names deletion and swapping as noise [arxiv:2104.06979 §3.1], but the released code implements only deletion [code: sentence_transformers/sentence_transformer/datasets/denoising_auto_encoder.py#L47-L49]. The paper's comparison of five noise types is the only evidence for the others [arxiv:2104.06979 app. A] [arxiv:2104.06979 tab. 5]. - **Masking scope in GraphFormers.** The paper masks tokens of every input node [arxiv:2105.02605 §3.3]. The released code masks only the centre node [code: src/data_handler.py#L166-L168]. - **Replacement rule in DRAGON.** The paper says only that a fraction of tokens is masked [arxiv:2210.09338 §2.3]. The released code uses whole-word masking by default, with 80% mask, 10% random and 10% kept [code: dragon.py#L723] [code: utils/data_utils.py#L181-L185]. - **Masking unit and rate in GottBERT.** The paper gives neither, and the card says not to fill the gap with RoBERTa defaults unless you label them as your own assumptions [arxiv:2012.02110 §3] [arxiv:2012.02110 §5]. - **Masking in data2vec text.** The paper's main text uses BERT masking, while the released config uses span masking, and the card places the config with a GLUE row rather than the main row [arxiv:2202.03555 §4.3] [arxiv:2202.03555 tab. 4] [code: examples/data2vec/config/text/pretraining/base.yaml#L25-L26]. - **Special tokens in ALBEF.** The paper does not say whether [CLS] or [SEP] may be masked. The released code protects only padding and [CLS], so [SEP] can be masked [code: models/model_pretrain.py#L257-L258]. - **Head sharing in TabTransformer-MLM.** The paper describes a per-column classifier for the replaced-value variant, but says only "a multi-class classifier" for MLM [arxiv:2012.06678 §2] [arxiv:2012.06678 app. A]. - **Averaging in StereoSet's masked score.** The card records that the paper averages log probabilities and the code averages raw probabilities [arxiv:2004.09456 §7.1] [code: code/eval_discriminative_models.py#L176-L177]. - **Masking rate in BETO.** The paper gives neither the masking rate nor the replacement rule [arxiv:2308.02976 §3]. - **Replacement rule in TaBERT.** The paper does not state the 80/10/10 split for context masking [arxiv:2005.08314 §3.2]. - **Link sampling in LinkBERT.** The paper does not state the segment length, how a segment is picked from a linked document, or what an anchor with no outgoing link does [arxiv:2203.15827 §4.2] [arxiv:2203.15827 §4.3]. The 1/in-degree normalisation is inferred, not stated. - **Masking rate in the ad-headline model.** The card states it is not given [arxiv:2607.06818 §3.1]. Its sampled-token stage also has no stated normalisation for the summed log-probability [arxiv:2607.06818 eq. 6]. - **Masking rate in FLASH's C4 MLM runs.** The card does not state it, and no code is released [arxiv:2202.10447 tab. 6]. - **Relation head in LinkBERT base and large.** These models start from BERT checkpoints whose head is next-sentence prediction. The paper does not say whether the three-class head starts fresh [arxiv:2203.15827 §5.1]. ## Contradictions Within the cards, the masking rules mostly agree. CamemBERT, FlauBERT and BEHRT mask units of the same kind, as tokens or codes. The label method and the cloze method use different units or no masked training at all. TSDAE deletes words rather than masking them, and the GraphFormers warm-up uses masks without a recovery loss. MoMask and MDLM both mask discrete units, but their schedules are different, and the cards do not compare them. The metrics also differ: tagging, parsing, NER and XNLI accuracy in CamemBERT, FLUE scores in FlauBERT, average precision and AUROC in BEHRT, mAP in the label method, precision at one in the cloze method, FID and R-precision in MoMask, bounded perplexity in MDLM, balanced accuracy in the partisan study, accuracy in Belebele, ARLUE and GLUE scores in the Arabic and DeBERTaV3 cards, a macro average over 35 tasks in FLAVA, and LAMBADA accuracy in GLM-130B. Their numbers should not be compared directly. CamemBERT's whole-word result and FlauBERT's subword setup are on different corpora and benchmarks, so the cards do not test the whole-word question for FlauBERT. Neither card settles it. Inside CamemBERT, the NER result in tab. 8 is higher for subword masking (91.55 vs 91.44), while the authors summarise the masking effect as small. The number 91.55 also appears for a different NER setting in tab. 5, which the card flags as an internal inconsistency. The CCNet BASE whole-word 100k model also appears with two different UPOS/LAS pairs in tab. 8 [arxiv:1911.03894 tab. 8]. Check the tables before quoting these values. TSDAE's advantage over MLM is not uniform. In the setting that trains on the target domain and then NLI+STS, the unsupervised averages are 56.5 for TSDAE and 55.9 for MLM, and MLM is ahead on AskUbuntu, 60.6 against 59.4 [arxiv:2104.06979 tab. 2]. The paper reports TSDAE ahead of MLM as a pretraining step on most datasets, with AskUbuntu as the exception [arxiv:2104.06979 §6]. For the cloze method, the main text says the headline results tune only layer 0. Table 1 matches the all-layer variant instead [arxiv:2104.06599 §4.7] [arxiv:2104.06599 tab. 5]. Inside MoMask, the HumanML3D FID for the full model is 0.045 in the main table [arxiv:2312.00063 tab. 1] and 0.051 in the ablation table [arxiv:2312.00063 tab. 2]. The KIT-ML value is also different across those tables. Cite the table you use. Inside MDLM, the text says the model approaches autoregressive perplexity within 14% on LM1B [arxiv:2406.07524 §5.1]. Against the table, the 327B-token bound is about 10% above the autoregressive baseline, and the 33B-token bound about 21% above [arxiv:2406.07524 tab. 1]. Recompute before quoting the percentage. TabTransformer's headline margin is measured with MLM. At 50 labels on the 8 larger datasets, MLM scores 1.2 points above the best other method, while the replaced-value variant scores 1.0 points above it [arxiv:2012.06678 tab. 3]. On the 12 smaller datasets, MLM trails the best other method at 50 and 200 labels [arxiv:2012.06678 tab. 4]. Read the margin claim as the better of the two objectives, not as a claim about MLM alone. BAE's headline claim is that its attacks lower accuracy more than TextFooler in most settings [arxiv:2004.01970 §3]. The paper's own tables disagree for one case: with BERT as the target and replacement only, TextFooler gives lower accuracy than BAE-R on six of seven datasets [arxiv:2004.01970 tab. 1] [arxiv:2004.01970 tab. 2]. The text says TextFooler is sometimes stronger on BERT [arxiv:2004.01970 §3]. Use the tables for BERT targets. Two paper and code pairs disagree on the masking rule. data2vec's paper uses BERT masking in its main text, while the text config uses spans with no random or unmasked tokens [arxiv:2202.03555 §4.3] [code: examples/data2vec/config/text/pretraining/base.yaml#L25-L26]. StereoSet's paper averages log probabilities for the masked score, and its code averages raw probabilities [arxiv:2004.09456 §7.1] [code: code/eval_discriminative_models.py#L176-L177]. BERTje's paper and released code disagree on the masking unit and rate. The paper hides consecutive wordpieces of the same word, at 15% of tokens [arxiv:1912.09582 §2.2]. The code keeps each wordpiece as a separate candidate, and caps predictions at 20 per sequence, so a full-length text is masked at about 4% [code: pretraining/create-pretrain-data.py#L32] [code: pretraining/create-pretrain-data.py#L34]. The card's advice is to decide which rule you mean before you train. ChemBERTa-2's comparison of MLM and MTR is on SMILES strings, not text. It does not test the text variants above. MTR is ahead on all eight Table 1 columns at 10M and 77M, but on six of eight at 5M, where MLM is ahead on BACE RMSE and BACE ROC [arxiv:2209.01712 tab. 1]. Results from the new cards are not directly comparable with the rest. The RoBERTa gain of +3 to +4 points is for a bundle of changes [arxiv:2103.11943 p. 12]. LinkBERT's MRQA scores use a dev split that the authors divided in two, so they are not leaderboard numbers [arxiv:2203.15827 §5.2]. ARBERT's ARLUE scores are on test splits [arxiv:2101.01785 §5.2], while DeBERTaV3's GLUE numbers are dev-set results, with the fine-tuning grid chosen on dev [arxiv:2111.09543 §6.4]. The ad-headline ROUGE-L gains are on proprietary campaign data, against the paper's own bi-LSTM baseline [arxiv:2607.06818 §4.2]. FLASH's speed-ups are training-cost comparisons on TPU-v4, against Transformer baselines that the authors re-ran with their own code [arxiv:2202.10447 §4.2]. ## Papers - [Ad Headline Generation using Self-Critical Masked Language Model](/papers/2607.06818): BERT-Large trained with masked-token loss on headline tokens, then tuned with self-critical policy gradient and ROUGE-L as reward. - [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](/papers/2107.07651): a fusion encoder with MLM and momentum distillation, after a contrastive alignment step. - [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](/papers/2101.01785): two BERT-Base encoders for Arabic, one on formal text and one on tweets, with the ARLUE benchmark. - [BAE: BERT-based Adversarial Examples for Text Classification](/papers/2004.01970): black-box word substitutions and insertions proposed by BERT's masked LM, filtered by sentence similarity and part of speech. - [BEHRT: Transformer for Electronic Health Records](/papers/1907.09538): a BERT-style Transformer over diagnosis histories, pretrained with masked disease prediction. - [BERT: A Review of Applications in Natural Language Processing and Understanding](/papers/2103.11943): survey of BERT's pretraining and uses, with MLM background, dynamic masking and translation language modeling. Its figures are taken from cited papers. - [BERTje: A Dutch BERT Model](/papers/1912.09582): BERT-base trained on Dutch text only, with sentence order prediction and a masking rule that the paper and code state differently. - [BOOM](/papers/2505.01912): a benchmark that holds out low-density molecules for property prediction; cited here as a weak link only. - [CamemBERT: a Tasty French Language Model](/papers/1911.03894): RoBERTa-style French encoder trained with whole-word masking; the ablation isolates masking granularity. - [ChemBERTa-2: Towards Chemical Foundation Models](/papers/2209.01712): RoBERTa-style encoder over SMILES, pretrained with MLM or with regression on RDKit descriptors, compared on MoleculeNet. - [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](/papers/2202.03555): one self-distillation recipe for speech, images and text, with a masked student that regresses teacher features. - [DeBERTaV3: ELECTRA-style pretraining with gradient-disentangled embedding sharing](/papers/2111.09543): replaces the masked-token loss with replaced-token detection, and shares embeddings with a stop-gradient on the generator side. - [Deep Bidirectional Language-Knowledge Graph Pretraining](/papers/2210.09338): an encoder pretrained with MLM and graph link prediction on text and a local knowledge-graph subgraph. - [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](/papers/2407.01392): per-token noise levels for sequence models; cited for its framing of masking as partial noise. - [Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](/papers/2504.08165): shared-task report whose grammar evaluation scores masked models by pseudo-likelihood. - [FlauBERT: Unsupervised Language Model Pre-training for French](/papers/1912.05372): BERT-style French encoder with subword MLM on 71 GB of text, evaluated on the FLUE benchmark. - [FLAVA: A Foundational Language And Vision Alignment Model](/papers/2112.04482): image, text and fusion encoders trained with masked multimodal, masked image, masked language and contrastive losses on public image-text pairs. - [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](/papers/2305.08283): probes encoder and decoder models for political lean, and measures how partisan pretraining changes hate-speech and misinformation classifiers. - [General Multi-label Image Classification with Transformers](/papers/2011.14027): hides labels in a Transformer classifier and predicts them, so known labels can be given at test time. - [GLM-130B: An Open Bilingual Pre-trained Model](/papers/2210.02414): a 130B-parameter English and Chinese model trained by blank infilling, with stability fixes and INT4 weight quantization. - [GottBERT: a pure German Language Model](/papers/2012.02110): RoBERTa-style German encoder pretrained on filtered and unfiltered German OSCAR. - [GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph](/papers/2105.02605): nests graph attention inside a BERT-style encoder, with a masked warm-up stage before fine-tuning. - [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](/papers/2305.14456): Arabic prompts with masked entities, scored by a cultural bias measure. - [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](/papers/2104.06599): learns continuous cloze prompts for querying a fixed masked language model. - [LinkBERT: Pretraining Language Models with Document Links](/papers/2203.15827): MLM on two-segment inputs built from document links, with a three-way relation loss in place of next-sentence prediction. - [MoMask: Generative Masked Modeling of 3D Human Motions](/papers/2312.00063): masked transformer over the base layer of a residual quantizer, then a residual transformer, for text-to-motion generation. - [Simple and Effective Masked Diffusion Language Models](/papers/2406.07524): trains an encoder to recover masked tokens under a weighted masked-token loss, and samples text by unmasking from an all-mask start. - [Spam and Sentiment Detection in Arabic Tweets Using MARBERT Model](/papers/2606.25495): fine-tunes the MARBERT encoder for five-class sentiment and for spam on private customer tweets, with class imbalance as the main difficulty. - [Spanish Pre-trained BERT Model and Evaluation Data](/papers/2308.02976): BERT-Base-sized encoder pretrained on Spanish text only, with whole-word and dynamic masking, and a Spanish evaluation suite. - [StereoSet: Measuring stereotypical bias in pretrained language models](/papers/2004.09456): context association tests with a stereotype, an anti-stereotype and an unrelated option, scored on language modeling and stereotype preference. - [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](/papers/2005.08314): BERT-initialised encoder over sampled table rows, pretrained with MLM and masked column and cell objectives. - [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](/papers/2012.06678): Transformer over categorical columns, with masked or replaced-value pretraining for semi-supervised use. - [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](/papers/2308.16884): a multiple-choice reading test, parallel across 122 variants, used to evaluate masked encoders and LLMs. - [Transformer Quality in Linear Time](/papers/2202.10447): gated attention unit with chunked exact and linear attention, evaluated on C4 MLM and LM at long context. - [TSDAE](/papers/2104.06979): sentence embeddings from deletion-based reconstruction, compared with MLM as an unsupervised pretraining step.