Relative positional encoding

Overview

Self-attention by itself ignores token order, so a Transformer needs some signal about where each token sits. The absolute approach gives every index its own vector and adds it to the token embedding. The relative approach leaves the input alone and lets the attention score between positions i and j depend on how i and j relate to each other, such as their offset or whether they share a segment. A model that uses only this kind of term sees relationships between positions, never the raw indices 1906.08237 §2.5.

Thirteen cards use a relative term in some form, and they differ in what it depends on. XLNet takes the relative positional scheme and segment recurrence from Transformer-XL (arXiv:1901.02860, no card here) and carries the relative idea over to segment identity 1906.08237 §2.4 1906.08237 §2.5. The XLNet card does not restate Transformer-XL's exact score decomposition. It only says that each layer applies "attention with relative positional encoding" (RelAttn) inside a post-LayerNorm block 1906.08237 app. A.2. To implement the positional term itself you need the Transformer-XL paper or the XLNet code (https://github.com/zihangdai/xlnet), and the card has not checked that code. ConveRT adds a learned bias per relative offset, capped per layer, on top of absolute position tables 1911.03688 §2.1. Tranception drops its position embeddings and adds a distance penalty per head group 2205.13760 §3.2. PaLM uses RoPE, which its card names as an alternative to relative embeddings 2204.02311 §2. Toto uses rotary positions with XPOS in its time-wise attention blocks 2505.14766 §3.1. Four vision cards use a learned bias over relative offsets inside local windows: Swin Transformer with a table indexed by offset, Video Swin Transformer with the same table over time, height and width, Swin Transformer V2, which replaces the table with a small network applied to log-scaled offsets, and FAST-VQA, which keeps two tables and picks one per pair depending on whether the two tokens lie in the same mini-patch of a spliced frame 2207.02595 §3.2. Voxel Transformer works on sparse 3D voxels and maps the coordinate difference between a query voxel and each attending voxel through a learned linear layer, adding the result to keys and values 2109.02497 §3.2 eq. 2. iRPE adds a learned bucket term over the row and column offset between two patches to ViT attention, on top of DeiT's absolute table 2107.14222 §3.2 §4.1. DocFormer adds clipped 1D token-offset terms to a multi-modal document encoder, next to box-based spatial terms 2106.11539 §3.1 eq. 6. DeBERTaV3 keeps the DeBERTa backbone, which uses disentangled attention with relative positions 2111.09543 §2.2 §3.1, but its card gives no formula for that term.

The property the relative variants share is that the term depends on how two positions relate. XLNet adds one requirement: the relation must be computed from original sequence positions, not from processing order or segment boundaries. XLNet uses this in two places, described below.

Variants

Relative positional encoding inside permutation language modeling (XLNet)

XLNet trains along a randomly sampled factorization order z, but it never shuffles the tokens. They stay in natural order with the encodings of their original positions, and z is enforced only through attention masks 1906.08237 §2.2. Applying relative positional encodings computed from the original sequence is then simple 1906.08237 §2.4. Because order is never moved, the distances a model sees in pretraining match the natural-order text it sees at fine-tuning 1906.08237 §2.2.

In XLNet's query stream, relative encoding is also where target position enters. The query stream starts from one shared learned vector w, not a position embedding, so the position it is meant to predict reaches it only through relative attention 1906.08237 §2.3 1906.08237 §2.4. The card notes that the paper does not explain how the relative term is built for the query stream's own position. Check the code before implementing.

Card: XLNet: Generalized Autoregressive Pretraining for Language Understanding.

Relative encodings with segment-level memory (Transformer-XL style, as used by XLNet)

The content states of the previous segment are cached per layer and joined to the keys and values of the current segment, KV = [\tilde{\mathbf{h}}^{(m-1)}, \mathbf{h}_{\mathbf{z}_{\le t}}^{(m-1)}] 1906.08237 §2.4. Positional encodings depend only on actual positions, so once the cached states exist this update no longer depends on the factorization order used for the previous segment 1906.08237 §2.4. The query stream reads memory the same way 1906.08237 §2.4. Memory is reused only when the previous segment comes from the same context 1906.08237 §2.5.

Relative segment encodings (XLNet)

BERT adds an absolute segment embedding to every token. XLNet instead encodes only whether two positions are in the same segment 1906.08237 §2.5:

• each attention head has learnable vectors \mathbf{s}_+ (same segment) and \mathbf{s}_- (different segment), plus a learnable bias \mathbf{b};
• when i attends to j, take \mathbf{s}_{ij} \in \{\mathbf{s}_+, \mathbf{s}_-\} and compute a_{ij} = (\mathbf{q}_i + \mathbf{b})^\top \mathbf{s}_{ij};
• add a_{ij} to the ordinary attention score.

The authors give two reasons: the inductive bias of relative encodings helps generalization (a claim they credit to Transformer-XL), and the model can be fine-tuned on inputs with more than two segments, which absolute segment embeddings do not allow 1906.08237 §2.5.

Learned relative bias with a per-layer span cap (ConveRT)

ConveRT's shared six-block Transformer uses single-headed attention. In each layer, a learned bias that depends only on relative position is added before the softmax. Scores for pairs beyond the layer's maximum distance are zeroed. The maximum distances are 3, 5, 48, 48, 48 and 48, from the first layer up 1911.03688 §2.1. The same encoder also has two learned absolute position tables, of 47 and 11 rows. The relative bias therefore adds to absolute encodings and does not replace them 1911.03688 §2.1.

Card: ConveRT: Efficient and Accurate Conversational Representations from Transformers.

Grouped ALiBi distance penalty (Tranception)

Tranception removes learned and sinusoidal position embeddings. Each head's query-key score gets a term that grows with distance, following ALiBi, but the slopes are set per head group so that each group can attend at a different range 2205.13760 §3.2.

The card's code check finds a gap here. The code computes slopes for one quarter of the heads and repeats that list for all four groups. The groups therefore share slope values and differ only in their convolution [see the Tranception card's Paper vs code section]. The card does not say which version produced the reported numbers. If you reimplement the paper, the group-wise ranges are the paper's claim, and the released code does something else.

Card: Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval.

Learned offset table inside shifted windows (Swin)

Swin computes attention only inside local M × M windows. Each head adds a learned bias B to the scaled query-key score before the softmax 2103.14030 §3.2 eq. 4. B is read from a table with (2M−1) × (2M−1) entries, one for each relative offset along each axis, so the bias depends only on where two tokens sit relative to each other inside the window 2103.14030 §3.2. The paper states that this bias beats having no position encoding and beats absolute position embeddings. It also states that adding absolute embeddings to the input slightly hurts detection and segmentation 2103.14030 §3.2 2103.14030 §4.4 tab. 4. The card does not carry those ablation numbers, so check the paper's table before you quote one.

Because the table is sized for one window, a model pretrained at one window size and run at another has to resize it. Swin V2 describes bicubic interpolation of the table as the usual fix and reports that it transfers poorly 2111.09883 §3.1 tab. 1.

Card: Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.

Three-axis offset table (Video Swin)

Video Swin applies the same mechanism to 3D windows of P × M × M tokens, with P = 8 and M = 7 by default 2106.13230 §3.2 §3.3. The paper's text writes the bias table as P² × M² × M² entries. The code uses one entry per relative offset in time, height and width, which is (2P−1)(2M−1)(2M−1) per head 2106.13230 §3.2. The card treats the text as a typo, and so should you.

To start from a 2D Swin checkpoint, the authors repeat the 2D table over the 2P−1 temporal offsets, so the bias is the same in every frame pair 2106.13230 §3.4. Their initialisation ablation compares this with a centre initialisation and finds the same Kinetics-400 top-1 for both, 78.8 2106.13230 tab. 9. The card notes that the centre variant is not in the backbone code, so that comparison cannot be reproduced from the released backbone.

Card: Video Swin Transformer.

Log-spaced continuous bias from a meta network (Swin V2)

Swin V2 replaces the table with a small meta network G, a two-layer MLP with a ReLU between the layers. G takes the relative offset (Δx, Δy) and returns one bias per head. Before G sees the offset, each coordinate is mapped to sign(x) · log(1 + |x|) 2111.09883 §3.3 eq. 4. Because G accepts any offset, a pretrained model can run at a larger window without a table to resize. For inference the biases can be computed once and stored as ordinary parameters 2111.09883 §3.3.

The reported gains are for Swin-T with a changed window, with and without fine-tuning. Without fine-tuning at window 24², the table gets 68.7 top-1 and the log-spaced bias gets 79.1 2111.09883 tab. 1. The text gives 78.9 for this cell, and the card uses the table value. With fine-tuning the same comparison is 83.2 and 84.2. At window 32² the log-spaced bias gives 51.8 box AP on COCO (50.9 for the table) and 47.8 mIoU on ADE20K (44.5 for the table) 2111.09883 tab. 1. The authors credit the gains to the log-spaced bias, and no control run isolates it 2111.09883 §4.2.

The released code departs from the formula in three places. It rescales the offsets by the window size before applying the log, and it uses a base-2 log with a normalisation. It also bounds each bias to (0, 16) with a scaled sigmoid. It recomputes the bias on every forward pass instead of caching it. The card's worked example in §3.3 follows the paper's formula, not the code. The card cannot tell which version produced the reported numbers 2111.09883 §3.3 [paper vs code].

Card: Swin Transformer V2: Scaling Up Capacity and Resolution.

Two tables selected by a same-patch gate (FAST-VQA)

FAST-VQA scores video quality on fragments: full-resolution mini-patches spliced onto one canvas per frame. Two tokens can share the same offset inside the canvas and still be far apart in the original frame, if they lie in different mini-patches. The backbone, a Video Swin-T, therefore learns two tables instead of one. A table T^real serves pairs inside one mini-patch, and T^pseudo serves pairs that cross mini-patches. A binary gate G is 1 for same-patch pairs and 0 otherwise, and the bias is added as G times the real term plus (1 minus G) times the pseudo term 2207.02595 §3.2 eq. 5 eq. 6.

The paper does not state the table shape or the index mapping, so this variant cannot be sized from the card. The authors' repository is under a non-commercial license, and the card does not use its code, so take the table layout from the paper's text and state the choice you make.

The ablation that removes this term is the cleanest test of a relative bias on this page. Removing GRPB from FANet gives 0.873 / 0.872 (SRCC / PLCC) on LSVQ test, against 0.876 / 0.877 for the full model. On LSVQ 1080p the figures are 0.769 / 0.805 against 0.779 / 0.814 2207.02595 tab. 7. The 1080p drop is 0.010 SRCC and 0.009 PLCC (derived). A variant that keeps GRPB only in the first two layers scores 0.873 / 0.875 and 0.772 / 0.809 tab. 7. The card records no run spread for these rows.

Card: FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling.

Learned linear map of 3D coordinate differences (Voxel Transformer)

VoTr applies self-attention only over non-empty voxels. For a query voxel i and an attending voxel j, the positional term is E_pos = (p_i − p_j) W_pos, a linear map of the difference between voxel centres. E_pos is added to both the key and the value 2109.02497 §3.2 eq. 1 eq. 2. The attending set is built from local and dilated ranges, and the paper caps it at 48 voxels per query 2109.02497 §3.3 §4.1. Unlike the Swin variants, the term has no table and no fixed window: it is computed from the actual metric offsets between voxels.

The card records no ablation that removes E_pos. Table 4 compares local attention alone with local plus dilated attention, and the other ablations vary dropout, the attending-voxel cap and model size 2109.02497 tab. 4 tab. 5 tab. 6. The positional scheme is therefore not isolated by any reported result.

Card: Voxel Transformer for 3D Object Detection.

Bucketed 2D offset term with a Product mapping (iRPE)

iRPE gives each pair of patches a bucket from its row offset and column offset together, and reads a learned term for that bucket. The index is exact for small offsets, log-spaced beyond a threshold α up to a cap β, and then rounded 2107.14222 §3.2 eq. 18. The paper compares four ways to map a 2D offset to a bucket. The Product mapping, with one bucket per pair of row and column buckets, gives the highest top-1 of the contextual variants it tests on DeiT-S 2107.14222 tab. 1. Its table has (2β+1)² entries §3.2. The contextual form dots each query with a learned vector per bucket. The dot products are computed once per bucket and then gathered for every pair, which keeps the extra cost small 2107.14222 §3.2 [eqs. 26–28]. The default places the term on keys only. The paper also tests queries and values 2107.14222 §4.1 tab. 4.

On DeiT-S, the keys-only term gives 80.9 top-1 on top of the absolute table, against 79.9 without it, and all three placements give 81.4 2107.14222 tab. 5. These are author-reported. The paper states run counts for one ablation only, so the other numbers have no stated spread. The released code truncates β to an integer and rounds distances before indexing, and its config names do not match the bucket counts; see Contradictions.

Card: Rethinking and Improving Relative Position Encoding for Vision Transformer.

Clipped token-offset terms next to box geometry (DocFormer)

DocFormer scores each branch of its attention with four terms: content, a relative query term and a relative key term over learned token-offset embeddings, and a spatial term 2106.11539 §3.1 eq. 6 eq. 7. The offset embeddings are clipped to 8 tokens on each side, and the paper does not say what happens beyond that [supp. §6.1]. The spatial term uses one pair of query and key matrices for both the visual and the text branch. Its inputs come from box corners and sizes, differences between neighbouring words, and an absolute 1D position embedding 2106.11539 §3.1 eq. 1 eq. 2.

No ablation removes the 1D relative terms alone. The architecture ablation adds spatial inputs and then the multi-modal attention as a whole, so the gain cannot be assigned to the relative terms 2106.11539 tab. 9. These ablations use 1M pre-training pages, not the 5M of the headline runs §4.5.

Card: DocFormer: End-to-End Transformer for Document Understanding.

Disentangled attention with relative positions (DeBERTa backbone)

DeBERTaV3 uses DeBERTa's architecture: disentangled attention and relative positions 2111.09543 §2.2 §3.1. The DeBERTaV3 paper changes the pre-training objective and how embeddings are shared, not the attention itself 2111.09543 §3.3. The card does not give the score formula, the distance buckets or the head-level details, so this page cannot describe how the relative term is computed. The original DeBERTa paper (arXiv:2006.03654) has no card here. Take the formula from that paper or from the code. The card records the microsoft/DeBERTa repository as MIT-licensed, but it does not cite the lines that implement the relative term.

Card: DeBERTaV3.

Rotary position embeddings (RoPE), as an alternative

PaLM uses RoPE and describes the choice as a replacement for absolute or relative position embeddings 2204.02311 §2. The PaLM card gives neither the RoPE base nor the fraction of head dimensions rotated, and it has no ablation of the choice 2204.02311 §2.

Toto, a decoder-only forecaster for observability metrics, uses rotary positions with XPOS in each transformer block 2505.14766 §3.1. Its ablation removes causal scaling, the single-distribution head, the robust loss and variate-wise attention, but not the positional scheme 2505.14766 app. E. There is no comparison against an absolute or relative alternative.

RoPE itself (arXiv:2104.09864) has no card here, so this page does not describe how it works.

When to use which

Three cards have an on/off ablation of the relative term. ConveRT removes its bias, and iRPE adds the term while keeping the learned absolute table (Table 4, row 1 against rows 8 to 12) 2107.14222 tab. 4. FAST-VQA removes its gated tables and the rest of the network stays the same 2207.02595 tab. 7. Swin V2 varies the form of the bias, not whether it exists. The XLNet, Tranception, PaLM and Swin evidence changes the term along with other parts of the model, and Video Swin's ablations change how the table is initialised rather than whether it exists. VoTr's ablations do not touch E_pos at all. DocFormer's ablation changes the whole multi-modal attention block, not the relative terms alone.

What the evidence supports, and what it does not:

• XLNet's ablation shows that switching from BERT-Base to a denoising autoencoder on Transformer-XL raises RACE from 64.3 to 65.03 and SQuAD2.0 F1 from 76.30 to 79.56 1906.08237 tab. 6. That swap changes the whole backbone (relative encoding, memory and more), so the gain cannot be credited to relative encoding by itself.
• Removing memory from XLNet-Base lowers RACE from 66.66 to 65.55 1906.08237 tab. 6. That measures memory, which relative encoding makes possible, and not the positional scheme itself.
• In XLNet-Large, the authors see attention patterns defined by relative position: heads that attend at a fixed stride relative to the query, and heads that mask out the relative right half. They suggest the relative attention mechanism is what makes these patterns possible 1906.08237 app. A.6. This comes from inspecting attention maps, not from a controlled test.
• ConveRT's ablation removes only the relative bias, and Reddit single-context R100@1 falls from 68.2 to 67.8. Removing the span ramp gives 67.7 1911.03688 tab. 4. The card records no variance, so the 0.4 gap is not established as more than noise.
• Tranception's ablation puts a GPT-2 baseline (0.324), a Primer baseline (0.314) and Tranception S (0.344) on the same validation subset, without retrieval 2205.13760 app. B.1. The GPT-2 baseline uses GELU and Tranception uses squared ReLU, so the gap is not the position scheme alone.
• iRPE's Table 4 keeps the learned absolute table and varies only the relative term on DeiT-S: 79.9 with no relative term, 80.9 with keys only, and 81.4 with queries, keys and values 2107.14222 tab. 4. Table 2 gives a run spread of ±0.04 to ±0.16 for one comparison 2107.14222 tab. 2, and the paper states no run count for Table 4. The 1.0 and 1.5 point gaps are larger than that spread. The 0.1 to 0.3 point rows are not distinguishable from it.
• Table 4's rows 2 to 7 drop the absolute table. Relative keys alone reach 80.9 there, against 79.9 for the learned absolute table 2107.14222 tab. 4. The same substitution does not hold for DETR: with no absolute encoding, the contextual term gets 38.7 AP against 39.5 for the sinusoidal baseline, and both together reach 40.8 2107.14222 tab. 6.
• Swin V2's table comparisons (above) vary the form of the bias for a changed window, not whether a bias exists. They come from one backbone, Swin-T, and the authors did not align the other settings across rows 2111.09883 §4.2.

Implementation sketch

This is the part the XLNet card states. The positional term inside rel_score follows Transformer-XL and the card does not specify it.

Inputs: current segment x (natural order, original positions p_cur),
        cached content states mem per layer (positions p_mem, same
        context only), segment ids seg, optional factorization mask M
For each layer m:
  # memory joined to keys/values                       [arxiv:1906.08237 §2.4]
  KV_h = concat(mem[m-1], h[m-1])
  # original positions, never permuted
  KV_pos = concat(p_mem, p_cur)
  for each head, query i, key j:
    # content term plus Transformer-XL positional term; the latter is
    # not on the card                                  [arxiv:1906.08237 §2.4]
    score[i,j] = content_score(q_i, k_j)
               + rel_pos_score(q_i, pos_i - pos_j)
    # relative segment term                            [arxiv:1906.08237 §2.5]
    score[i,j] += (q_i + b)^T (s_plus if seg_i == seg_j else s_minus)
    # permutation order lives only in M                [arxiv:1906.08237 §2.2]
    mask out j where M forbids it
  # post-LayerNorm block                               [arxiv:1906.08237 app. A.2]
  h[m] = LayerNorm(h[m-1] + RelAttn(...))
  h[m] = LayerNorm(h[m] + PosFF(h[m]))
# content stream only is cached                        [arxiv:1906.08237 §2.4]
After the segment: mem[m] = h[m]

Where variants differ:

• XLNet's query stream uses the same layer, with w as its initial state and keys/values taken from content states at strictly earlier positions in z 1906.08237 §2.3 1906.08237 app. A.2.
• ConveRT replaces rel_pos_score with a learned scalar per relative offset and layer, and sets scores beyond that layer's span to zero 1911.03688 §2.1. The card gives the index as n − i + j and does not define n.
• Tranception replaces rel_pos_score with a distance penalty that grows with |i − j|, using one slope per head group, and adds no position embedding to the input 2205.13760 §3.2. The card states this follows ALiBi. The slope values are the open point, as described under the variant above.
• Swin replaces rel_pos_score with a lookup into a table of (2M−1)² values per head, indexed by the offset along each axis, and adds the shifted-window mask on the shifted blocks 2103.14030 §3.2 eq. 4.
• Video Swin uses the same lookup over three axes, with (2P−1)(2M−1)(2M−1) entries per head in the code 2106.13230 §3.2.
• FAST-VQA replaces rel_pos_score with a gated choice between two lookups: gate * real_table[offset] + (1 - gate) * pseudo_table[offset], where gate is 1 for tokens in the same mini-patch 2207.02595 §3.2 eq. 5 eq. 6. The table shape and index mapping are not stated, so offset is a placeholder.
• Swin V2 replaces the lookup with G applied to log-scaled offsets, one output per head 2111.09883 §3.3 eq. 3 eq. 4. The code's variant differs as described under the variant above.
• VoTr replaces rel_pos_score with E_pos = (p_i − p_j) W_pos, added to keys and values before attention. Query voxels in sparse modules are max-pooled from their attending features, and the positional term is not added to the query 2109.02497 eq. 1 eq. 2 eq. 4.
• RoPE (PaLM) changes the positional term. The PaLM card does not describe that change 2204.02311 §2.
• Toto applies rotary positions with XPOS in its transformer layers 2505.14766 §3.1. The card does not describe the XPOS formula, so take it from the code before implementing.
• iRPE replaces rel_pos_score with a table lookup. Each (row, column) offset pair maps to a bucket through the piecewise index, and there is one table per enabled position (queries, keys or values). The key-side term is added to the logits before the division by √d, and the value term is added to the output. The class token takes one extra bucket 2107.14222 §3.2 eq. 13 eq. 15 eq. 17 eq. 18 §4.1.
• DocFormer replaces rel_pos_score with two learned offset terms, one on the query side and one on the key side, with offsets clipped to 8 tokens each side. It adds a spatial score computed from query and key matrices shared between the visual and text branches 2106.11539 §3.1 eq. 6 eq. 7 [supp. §6.1].

Contradictions

None known between the cards on the main claim. iRPE's introduction says earlier vision work disagreed on whether relative encodings beat absolute ones: one found no gain, one found a gain, and one found that relative encodings do not match absolute ones 2107.14222 §1. Those works have no card here, so this is iRPE's account only. ConveRT's ablation and XLNet's evidence measure different things (the bias alone versus a whole backbone), so their numbers cannot be compared. Tranception's paper and code disagree on whether head groups have different slope ranges. Swin V2's paper and code disagree on the coordinate transform and on bounding the bias output; Video Swin's text and code disagree on the table's shape. VoTr's paper and code also differ. The paper uses a linear map for E_pos eq. 2; the code uses a 1 × 1 convolution followed by ReLU, and takes the coordinate difference with the opposite sign 2109.02497 Paper vs code, items 4 and 12. These are paper-versus-code conflicts, not disagreements between papers, and each is described under its variant above. Swin V2's text and table also give different values for one cell (78.9 and 79.1), which is a conflict inside one paper; the page uses the table. iRPE's code applies the index to squared distances in its quantization mapping, where the paper takes square roots first 2107.14222 Paper vs code. The Table 1 config names do not match the bucket counts the code produces, and the DeiT and DETR READMEs report numbers that differ from Tables 5 and 6 by 0.1 to 0.2 points 2107.14222 tab. 5 tab. 6. DocFormer's equations and supplementary pseudocode disagree on whether each branch gets its own spatial input, and whether the offset table is shared across branches 2106.11539 §3.1 [supp. §6.3]. These are paper-versus-code conflicts; each is described under its variant above.

Papers

• XLNet: Generalized Autoregressive Pretraining for Language Understanding: uses Transformer-XL relative positional encoding computed from original positions so that permutation LM and segment memory work, and adds relative segment encodings.
• ConveRT: Efficient and Accurate Conversational Representations from Transformers: adds a learned per-offset bias with a per-layer span cap to absolute position tables; its one ablation removes only the bias.
• PaLM: Scaling Language Modeling with Pathways: uses RoPE in place of absolute or relative position embeddings, with no ablation.
• Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval: uses grouped ALiBi distance penalties in place of learned position embeddings; no ablation isolates the position scheme.
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows: learns a bias table over relative offsets inside shifted windows, in place of absolute position embeddings.
• Video Swin Transformer: extends the windowed offset table to time, height and width, and initialises it from a 2D Swin checkpoint.
• Swin Transformer V2: Scaling Up Capacity and Resolution: replaces the table with a log-spaced continuous bias from a small MLP, so the bias transfers across window sizes.
• FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling: adds a same-patch gate that selects one of two relative bias tables for tokens in a spliced fragment; removing the term lowers LSVQ 1080p scores.
• Voxel Transformer for 3D Object Detection: adds a learned linear map of coordinate differences between voxels to keys and values in sparse 3D attention; no ablation removes it.
• DeBERTaV3: keeps DeBERTa's disentangled attention with relative positions as its backbone; the card gives no formula for the relative term.
• This Time is Different: An Observability Perspective on Time Series Foundation Models: uses rotary positions with XPOS in a decoder-only forecaster; the positional scheme is not ablated.
• DocFormer: End-to-End Transformer for Document Understanding: adds clipped 1D token-offset terms and a shared spatial score to multi-modal self-attention; no ablation isolates the relative terms.
• Rethinking and Improving Relative Position Encoding for Vision Transformer: adds bucketed 2D offset terms to ViT and DETR attention with a Product mapping; for classification the relative term can replace the absolute table, but for DETR it cannot.