--- kind: concept slug: permutation-language-modeling title: Permutation language modeling summary: >- Pretrain autoregressively over a randomly sampled factorization order instead of left to right, so each token learns from context on both sides without [MASK] tokens or independence assumptions. The order lives only in the attention masks; a two-stream attention makes predictions aware of the target position. Trade-off: extra attention work and harder optimization versus BERT-style masking. tags: [pretraining, language-modeling, transformer] papers: ["1906.08237"] related: [language-model-pretraining, relative-positional-encoding] status: draft updated_at: 2026-10-18 --- ## Overview Left-to-right language modeling uses the exact chain rule but gives each position only one side of its context. BERT-style masked language modeling (a denoising autoencoder) sees both sides, but it reconstructs the masked tokens independently of one another and relies on a `[MASK]` symbol that never shows up at fine-tuning time [arxiv:1906.08237 §2.1]. Permutation language modeling keeps the chain-rule factorization and changes only its *order*. For each sequence it samples a permutation z of the positions and maximizes the likelihood of each token given the tokens that come before it in z. Parameters are shared across all orders, so in expectation every token is trained with context from both sides [arxiv:1906.08237 §2.2] [arxiv:1906.08237 eq. 3]. The invariant every implementation has to respect: **the input is never shuffled.** Tokens stay in natural order with their original positions, and the sampled order exists only in the attention masks. The reason is that fine-tuning feeds natural-order text [arxiv:1906.08237 §2.2]. Compared with masking, the objective covers strictly more dependencies. With targets T and non-targets N, a masked model learns log p(x | N) for each target, while the permutation model learns log p(x | N plus the targets earlier in z). For the targets "New York" given "is a city", only the permutation model gets a term that predicts "York" from "New" [arxiv:1906.08237 §2.6]. ## Variants ### XLNet: two-stream attention with partial prediction XLNet is the only variant with a card so far. It adds four pieces to the bare objective: - **Target-aware prediction.** With a plain Transformer and permuted masks, two orders that share a prefix but continue at different positions get the same predicted distribution, because the model cannot tell which position comes next [arxiv:1906.08237 §2.3] [arxiv:1906.08237 app. A.1]. XLNet therefore conditions the softmax on the target position [arxiv:1906.08237 eq. 4]. - **Two-stream self-attention.** Each position keeps two hidden states. The *content* stream behaves like ordinary self-attention and includes the token itself. The *query* stream starts from one shared learned vector, attends only to content states of strictly earlier positions in z, and produces the logits. Both streams share weights. At fine-tuning the query stream is dropped and the network is a normal Transformer-XL [arxiv:1906.08237 §2.3] [arxiv:1906.08237 app. A.2]. - **Partial prediction.** Predicting every token of z converged slowly, so only the last part of the order, about 1/K of the tokens, is used as targets. Query states are computed only for those targets. XLNet-Large uses K = 6 [arxiv:1906.08237 §2.3] [arxiv:1906.08237 tab. 7]. - **Span-based targets.** A span of L consecutive tokens, with L between 1 and 5, is chosen as the targets. The paper describes this in a single sentence [arxiv:1906.08237 §3.1]. It runs on a Transformer-XL backbone. Relative positions are computed from original positions, so cached memory from the previous segment can be reused without knowing that segment's order [arxiv:1906.08237 §2.4]. Reported effect: in a 12-layer ablation trained on BooksCorpus and Wikipedia, swapping a denoising-autoencoder objective for the permutation objective on the same Transformer-XL backbone improved RACE, SQuAD2.0 and MNLI for both K = 6 and K = 7. SST-2 improved with K = 6 and stayed about level with K = 7 [arxiv:1906.08237 tab. 6]. At Large size with BERT's data and hyperparameters, XLNet beat the best of three BERT-Large variants on all 11 tasks the authors compared [arxiv:1906.08237 tab. 1]. Card: [XLNet: Generalized Autoregressive Pretraining for Language Understanding](/papers/1906.08237). ## When to use which | Situation | Use | Why | |---|---|---| | Encoder for fine-tuned understanding tasks, and you want both-side context without `[MASK]` | Permutation LM with two-stream attention | Exact factorization, no pretrain/fine-tune input mismatch; beat a matched DAE baseline on RACE, SQuAD2.0 and MNLI in the ablation [arxiv:1906.08237 tab. 6] | | Tight pretraining compute or a simple implementation | BERT-style masking or plain left-to-right LM | Permutation LM needs two attention streams for target positions and per-sequence order masks; XLNet-Large took 512 TPU v3 chips for about 5.5 days and was still underfit [arxiv:1906.08237 §2.3] [arxiv:1906.08237 §3.1] | | Long-context downstream tasks | Permutation LM on a Transformer-XL backbone with memory | Removing memory hurt most on RACE, the longest-context task in the ablation [arxiv:1906.08237 tab. 6] | | Choosing K (prediction ratio) | Treat it as task-dependent; K = 6 is the published Large setting | K = 7 beat K = 6 on SQuAD2.0 and MNLI but lost on RACE and SST-2 [arxiv:1906.08237 tab. 6] | The page has a single source, so it cannot compare permutation LM against other bidirectional-AR objectives. Any such comparison would need further cards. ## Implementation sketch ```text per sequence x (natural order, length T), with memory from the previous segment: z = random permutation of positions 1..T # span-based target choice in XLNet c = cut so that about T/K positions follow it # targets = z[c+1 .. T] rank[z[t]] = t content_mask[i, j] = rank[j] <= rank[i] # sees itself and earlier-in-z query_mask[i, j] = rank[j] < rank[i] # never sees its own token (both masks also allow all memory positions) h = embed(x) # content stream, all positions g = shared_vector w at targets # query stream, targets only for each layer: h_new = Attn(Q=h, KV=[mem, h], mask=content_mask) # relative positions g_new = Attn(Q=g, KV=[mem, h], mask=query_mask) # same weights h, g = h_new, g_new loss = - sum over targets of log softmax(E · g)[x_target] memory = this segment's content states h per layer fine-tuning: drop g and query_mask; run the content stream as a normal Transformer ``` Mask construction, two-stream layer and partial prediction follow [arxiv:1906.08237 §2.3] and [arxiv:1906.08237 app. A.2]. Memory reuse follows [arxiv:1906.08237 §2.4]. The span sampling, memory length and "bidirectional data pipeline" are underspecified in the paper; the card lists them as items to check in the official code. Common mistakes: - Shuffling the tokens instead of the masks [arxiv:1906.08237 §2.2]. - Letting the query mask include the diagonal, which leaks the target and makes the loss trivial [arxiv:1906.08237 §2.3]. - Dropping the query stream during pretraining. A single stream cannot tell target positions apart [arxiv:1906.08237 app. A.1]. ## Contradictions None known. ## Papers - [XLNet: Generalized Autoregressive Pretraining for Language Understanding](/papers/1906.08237): introduces permutation LM pretraining with two-stream attention, partial prediction and a Transformer-XL backbone.