Permutation language modeling

Overview

Left-to-right language modeling uses the exact chain rule but gives each position only one side of its context. BERT-style masked language modeling (a denoising autoencoder) sees both sides, but it reconstructs the masked tokens independently of one another and relies on a [MASK] symbol that never shows up at fine-tuning time 1906.08237 §2.1.

Permutation language modeling keeps the chain-rule factorization and changes only its order. For each sequence it samples a permutation z of the positions and maximizes the likelihood of each token given the tokens that come before it in z. Parameters are shared across all orders, so in expectation every token is trained with context from both sides 1906.08237 §2.2 1906.08237 eq. 3.

The invariant every implementation has to respect: the input is never shuffled. Tokens stay in natural order with their original positions, and the sampled order exists only in the attention masks. The reason is that fine-tuning feeds natural-order text 1906.08237 §2.2.

Compared with masking, the objective covers strictly more dependencies. With targets T and non-targets N, a masked model learns log p(x | N) for each target, while the permutation model learns log p(x | N plus the targets earlier in z). For the targets "New York" given "is a city", only the permutation model gets a term that predicts "York" from "New" 1906.08237 §2.6.

Variants

XLNet: two-stream attention with partial prediction

XLNet is the only variant with a card so far. It adds four pieces to the bare objective:

• Target-aware prediction. With a plain Transformer and permuted masks, two orders that share a prefix but continue at different positions get the same predicted distribution, because the model cannot tell which position comes next 1906.08237 §2.3 1906.08237 app. A.1. XLNet therefore conditions the softmax on the target position 1906.08237 eq. 4.
• Two-stream self-attention. Each position keeps two hidden states. The content stream behaves like ordinary self-attention and includes the token itself. The query stream starts from one shared learned vector, attends only to content states of strictly earlier positions in z, and produces the logits. Both streams share weights. At fine-tuning the query stream is dropped and the network is a normal Transformer-XL 1906.08237 §2.3 1906.08237 app. A.2.
• Partial prediction. Predicting every token of z converged slowly, so only the last part of the order, about 1/K of the tokens, is used as targets. Query states are computed only for those targets. XLNet-Large uses K = 6 1906.08237 §2.3 1906.08237 tab. 7.
• Span-based targets. A span of L consecutive tokens, with L between 1 and 5, is chosen as the targets. The paper describes this in a single sentence 1906.08237 §3.1.

It runs on a Transformer-XL backbone. Relative positions are computed from original positions, so cached memory from the previous segment can be reused without knowing that segment's order 1906.08237 §2.4.

Reported effect: in a 12-layer ablation trained on BooksCorpus and Wikipedia, swapping a denoising-autoencoder objective for the permutation objective on the same Transformer-XL backbone improved RACE, SQuAD2.0 and MNLI for both K = 6 and K = 7. SST-2 improved with K = 6 and stayed about level with K = 7 1906.08237 tab. 6. At Large size with BERT's data and hyperparameters, XLNet beat the best of three BERT-Large variants on all 11 tasks the authors compared 1906.08237 tab. 1. Card: XLNet: Generalized Autoregressive Pretraining for Language Understanding.

When to use which

The page has a single source, so it cannot compare permutation LM against other bidirectional-AR objectives. Any such comparison would need further cards.

Implementation sketch

per sequence x (natural order, length T), with memory from the previous segment:
  z      = random permutation of positions 1..T     # span-based target choice in XLNet
  c      = cut so that about T/K positions follow it  # targets = z[c+1 .. T]
  rank[z[t]] = t
  content_mask[i, j] = rank[j] <= rank[i]    # sees itself and earlier-in-z
  query_mask[i, j]   = rank[j] <  rank[i]    # never sees its own token
  (both masks also allow all memory positions)

  h = embed(x)                       # content stream, all positions
  g = shared_vector w at targets     # query stream, targets only
  for each layer:
      h_new = Attn(Q=h, KV=[mem, h], mask=content_mask)   # relative positions
      g_new = Attn(Q=g, KV=[mem, h], mask=query_mask)     # same weights
      h, g  = h_new, g_new
  loss = - sum over targets of log softmax(E · g)[x_target]
  memory = this segment's content states h per layer
fine-tuning: drop g and query_mask; run the content stream as a normal Transformer

Mask construction, two-stream layer and partial prediction follow 1906.08237 §2.3 and 1906.08237 app. A.2. Memory reuse follows 1906.08237 §2.4. The span sampling, memory length and "bidirectional data pipeline" are underspecified in the paper; the card lists them as items to check in the official code.

Common mistakes:

• Shuffling the tokens instead of the masks 1906.08237 §2.2.
• Letting the query mask include the diagonal, which leaks the target and makes the loss trivial 1906.08237 §2.3.
• Dropping the query stream during pretraining. A single stream cannot tell target positions apart 1906.08237 app. A.1.

Contradictions

None known.

Papers

• XLNet: Generalized Autoregressive Pretraining for Language Understanding: introduces permutation LM pretraining with two-stream attention, partial prediction and a Transformer-XL backbone.