DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing

In plain words

Large language models learn by hiding words and guessing them back. This paper keeps that general setup but changes the guessing game. A small helper model fills the hidden slots with plausible but sometimes wrong words, and the main model has to point at each word and say whether it was original or swapped in. Every word gets a verdict, not just the hidden ones, so the main model learns from more of each text.

The helper and the main model normally share one table of word vectors. The paper finds that the two training goals pull those shared vectors in opposite directions, which slows learning. The fix is to let the main model read the shared table but never send its own learning signal back into it, so only the helper shapes those vectors. The authors report that the resulting model does better on language-understanding tests than comparable models.

TL;DR

• DeBERTaV3 is DeBERTa with its masked-token objective replaced by replaced token detection (RTD) from ELECTRA.
• Generator and discriminator share token embeddings, but the discriminator's copy receives no gradient into the generator's embeddings (gradient-disentangled embedding sharing, GDES).
• Author-reported: DeBERTaV3-Large averages 91.37 on the GLUE development set, and the multilingual mDeBERTaV3-Base reaches 79.8 zero-shot XNLI accuracy.
• Pre-training settings differ between Section 3 (small ablation runs) and Section 4 (full runs), and the paper's own text and tables disagree on several values. Check the Pitfalls section before reproducing.
• Code is MIT-licensed and the released RTD scripts use different hyperparameters from the paper.

Problem

Masked language modeling (MLM) trains only on the masked fraction of each input, which is about 15% of tokens §2.3.1. RTD trains the discriminator on every token, so it is more sample-efficient, but ELECTRA shares its token embeddings between generator and discriminator, and the paper argues that this sharing hurts the discriminator. The MLM loss wants semantically similar tokens close together, while the RTD loss wants them separable §3.2. The paper measures this with slower convergence and lower average cosine similarity in the discriminator's embeddings §3.2, tab. 1, fig. 2. Separate embeddings (no sharing) avoid the conflict but lose the benefit of sharing, so the discriminator fine-tunes worse §3.2, tab. 2.

Method

Overview. The model is DeBERTa's architecture (disentangled attention and relative positions) pre-trained with ELECTRA's generator and discriminator §3.1. The paper's contribution is the embedding-sharing rule, GDES §3.3.

1. Backbone. Use DeBERTa's disentangled attention and enhanced mask decoder; the RTD objective replaces DeBERTa's MLM §2.2, §3.1.
2. Generator. Same width as the discriminator, half its depth. It is trained with MLM on about 15% of input tokens replaced by [MASK] §3.1, §6.4.
3. Replacement. Each masked position receives a token sampled from the generator's output distribution, not its argmax eq. 3.
4. Discriminator. Predicts, for each position, whether the token is the original one. Trained with binary cross-entropy over all positions eq. 4.
5. Joint objective. The generator's MLM loss and the discriminator's RTD loss are combined with weight λ; the paper uses λ = 50, as in ELECTRA §2.3.2, §3.1.
6. GDES. The discriminator's token embedding is the generator's embedding with its gradient stopped, plus a separate residual matrix that is zero at initialisation and is the only part the discriminator trains §3.3:

E_D = \mathrm{sg}(E_G) + E_\Delta, \qquad E_\Delta \leftarrow 0 \text{ at init} \qquad [\S 3.3]

7. Schedule. In each iteration, generate the discriminator inputs, update the generator and its embeddings with the MLM loss, then update the discriminator with the RTD loss, where gradients reach the shared embeddings only through E_Δ §3.3.

Figure 1 shows the three gradient paths: ES (a), NES (b) and GDES (c) fig. 1. In the paper's words, GDES lets the generator's embedding be learned from MLM only, and the discriminator's embedding builds on it §3.3.

# One GDES iteration  [§3.3] [eq. 2] [eq. 3] [eq. 4]
# Input: masked batch, original ids, generator G, discriminator D
#        E_G (generator embedding), E_delta (zero-initialised residual)
# Output: updated G, D, E_delta

def gdes_iteration(batch, rtd_lambda):
    # 1. Generator forward pass and MLM loss  [§3.1] [eq. 2]
    gen_logits = G(batch.masked_ids)                # (B, L, V)
    mlm_loss = masked_cross_entropy(gen_logits, batch.original_ids)
    # 2. Update generator and E_G from MLM only  [§3.3]
    update(G, mlm_loss)
    # 3. Sample a replacement for each masked position  [eq. 3]
    sampled = sample_from_softmax(gen_logits[batch.mask])
    corrupted = replace_at(batch.original_ids, batch.mask, sampled)
    # 4. Build the discriminator embedding with stopped gradient  [§3.3]
    E_D = stop_gradient(E_G) + E_delta              # (V, d)
    # 5. Per-token RTD loss, weight lambda = 50 in the paper  [eq. 4]
    labels = corrupted != batch.original_ids        # True = replaced
    rtd_loss = binary_cross_entropy(D(corrupted, E_D), labels)
    update(D, rtd_lambda * rtd_loss)                # E_D grads -> E_delta
    return

Notes on the sketch: the code writes labels as "replaced" only where the sampled token differs from the original, which matches the indicator in equation 4 replaced_token_detection_model.py. The decoupled schedule that the released code actually runs is described under Paper vs code.

Hyperparameters

Values for the full pre-training runs in Section 4 are in tab. 10. Where the text disagrees with the table, both are listed in Pitfalls.

Results

Author-reported. Setup: GLUE development sets, one classification head on [CLS] §4.1.1; large models use DeBERTaV3 with 24 layers and hidden size 1024, compared with BERT, RoBERTa, XLNet, ELECTRA and DeBERTa of similar structure §4.1.1. Numbers are on dev sets, with per-task hyperparameter search selected on the same dev set §6.4.

GLUE, large models tab. 3

Other tasks, large models tab. 4

The paper's Table 4 also lists ALBERT-xxlarge and Megatron variants. Only the rows above are reproduced here; the paper says DeBERTaV3-Large outperforms ALBERT-xxlarge and Megatron-1.3B on these tasks §4.1.1.

Base and smaller models tab. 5

Embedding-sharing ablation, 125k-step setting tab. 1, tab. 2

Cosine similarity is averaged over a 10% sample of word pieces [tab. 1, footnote in §3.2]. The DeBERTa-base baseline in this setting is 86.3/86.2 on MNLI and 82.5/79.3 on SQuAD v2.0 tab. 2.

Multilingual XNLI averages tab. 6

GDES on ELECTRA-base, reimplemented tab. 9, §6.3

Paper vs code

Code is the microsoft/DeBERTa repository at commit 4d7fe0bd4fb3c7d4f4005a7cafabde9800372098, MIT licensed. The RTD implementation lives in DeBERTa/apps/tasks/rtd_task.py.

• Lambda. Paper: λ = 50 §2.3.2 §3.1. Code default: --rtd_lambda defaults to 10 rtd_task.py, and the RTD scripts do not pass it rtd.sh. Set it explicitly to match the paper.
• Scripts versus paper settings. Paper full runs use batch 8,192 and 500,000 steps with learning rate 3e-4 (large) or 5e-4/6e-4 (base and small) §4 tab. 10. The released rtd.sh targets the released DeBERTaV3 checkpoints with batch 256, learning rate 1e-4 for base and large, and --num_training_steps 1000000 rtd.sh. These are different runs; the script is not a replay of Section 4.
• Embedding sharing scope. Paper: GDES applies to token embeddings §3.3. Code also applies the same stop-gradient construction to the position embeddings rtd_task.py.
• GDES formula. Matches: the discriminator weight is the generator weight detached plus a trainable residual, computed in a forward pre-hook rtd_task.py.
• Decoupled schedule. The released default runs the generator and discriminator as two trainers. The discriminator takes one update per generator update on the same batch, after the generator step rtd_task.py. The paper describes the same alternation §3.3, but not a separate optimizer.
• Discriminator learning rate. The code can override the discriminator learning rate with --discriminator_learning_rate (default -1, meaning the same rate) rtd_task.py. The paper states no separate rate.
• Sampling. The code calls its sampler topk_sampling with topk=1. That is sampling from the full softmax at temperature 1, not argmax rtd_task.py, which matches equation 3. The name is misleading.
• Masking. Masked positions are replaced with [MASK] only: keep_prob=0 and mask_prob=1 rtd_task.py. The default masking budget is round(0.15 × length), capped by max_preds_per_seq mlm_task.py.
• Discriminator head. The code adds the [CLS] hidden state to every token before the dense and LayerNorm head replaced_token_detection_model.py. The paper does not describe this step.
• Architectures. Generator depth is half the discriminator depth in the configs, as the paper says rtd_base.json. Large uses 16 attention heads rtd_large.json, which disagrees with the table (see Pitfalls).

Pitfalls

• λ is not stated consistently. Paper uses 50 §2.3.2, §3.1. Code defaults to 10 and the released scripts do not override it rtd_task.py.
• Learning rate for base and small. Text says 5e-4 §4; the hyperparameter table says 6e-4 tab. 10. Pick one and record the choice.
• Large attention heads. The table gives 12 heads with head size 64 and hidden size 1024 tab. 10, which does not add up (1024 / 64 = 16). The code uses 16 heads rtd_large.json.
• Optimizer name and β2. Section 4 calls the optimizer AdamW with β2 = 0.98 §4; Section 6.4 calls it Adam with weight decay §6.4. The full-run column of Table 10 gives β2 = 0.98, and the Section 3 column gives 0.999 tab. 10.
• Two pre-training regimes. Section 3 ablations use Wikipedia and BookCorpus, batch 2,048, 125k steps §3.1. Section 4 uses 160 GB, batch 8,192, 500k steps §4. Table 2 numbers come from the smaller regime and are not comparable with Tables 3 to 6.
• Data size for mDeBERTa. The text says 2.5T §4.2; the appendix says 2.5TB §6.2.
• Vocabulary size. Paper: 128,000 tokens §4. Config: 128100 rtd_base.json. The released scripts use a SentencePiece model downloaded from Hugging Face rtd.sh.
• Model selection on dev. GLUE and SQuAD numbers are dev-set results, and the fine-tuning grid is selected on the same dev sets tab. 11 §6.4. Treat the margins as optimistic relative to a held-out test.
• Hidden cost. GDES adds a vocabulary-sized residual matrix E_Δ on the discriminator side rtd_task.py. The paper says the per-step computation is the same as ES and NES §3.3; it does not give the memory cost.
• Stored discriminator embedding. The code recomputes the discriminator embedding each forward pass and registers it as a buffer rtd_task.py. So a saved discriminator state holds the summed embedding, not the residual alone. This is an inference from the code; I did not run a checkpoint save.
• Figure reading. Figure 2 (the generator's MLM loss under the three sharing methods) is described only in prose. The paper's claim that NES converges faster than ES, and that GDES matches NES, comes from that figure §3.2, §3.3, fig. 2.

Lineage

• Builds on: DeBERTa (He et al., 2020) for disentangled attention and the enhanced mask decoder §2.2; ELECTRA (Clark et al., 2020) for replaced token detection and the embedding-sharing baseline §2.3.
• Compared against: BERT, RoBERTa, XLNet, ALBERT, Megatron, T5 (as context only, since its test results are not comparable) §4.1.1, footnote on T5; XLM, mT5 and XLM-R for multilingual models §4.2; TinyBERT and MiniLMv2 for smaller models tab. 5.