---
kind: concept
slug: positional-encoding
title: Positional encoding
summary: >-
  How attention, graph and generative models learn where each input sits, since attention alone ignores order. Variants: sinusoids added to the input, a flow condition on location, date and band codes, PPEG convolutions, rotary, distance and relative-bucket biases, box and camera-ray geometry, NeRF bands, an acyclic time code, and a learned node stream.
tags: [attention, transformer, position-encoding, architecture]
papers: ["2102.04942", "2103.15808", "2104.00674", "2104.00678", "2104.06405", "2104.10157", "2104.14786", "2105.13677", "2106.00908", "2107.12571", "2107.14222", "2110.07875", "2112.14683", "2201.07284", "2202.10447", "2205.01138", "2207.08051", "2211.05100", "2302.13971", "2308.12950", "2407.09777", "2409.19648", "2412.02732", "2412.04431", "2506.20380", "2512.07237", "2601.14724", "2112.05504", "2203.05328", "2311.17633"]
related: [relative-positional-encoding, linear-attention, self-attention, attention-mechanism, object-detection, object-queries, vision-transformer, convolutional-neural-network, time-series-forecasting, language-model-pretraining, neural-radiance-field, novel-view-synthesis, level-of-detail-rendering, visual-object-tracking, image-registration, kv-cache-management, multiple-instance-learning, anomaly-detection, code-completion, camera-pose-encoding, fourier-features, masked-image-modeling, normalizing-flows, self-supervised-learning, foundation-model, graph-neural-network, graph-transformer, molecular-representation-learning, neural-implicit-surface, physically-based-rendering, differentiable-rendering, generative-adversarial-network, video-generation, implicit-neural-representation]
status: draft
updated_at: 2026-11-01
---

## Overview

Attention compares every position with every other position, so by itself it
cannot tell which element came first. Positional encoding supplies that
information, or a substitute for it such as the box geometry of a detected
object. Each card here does it differently:

- **Fixed sinusoids added to the input embedding**, one vector per token index.
  This is the original Transformer scheme that the tutorials describe
  [arxiv:2205.01138 §2.5.1] [arxiv:2311.17633 §2.2]. Satellite and video models reuse the same sine and
  cosine form for each axis of a patch grid [arxiv:2412.02732 §3.2] [eq. 1].
- **Learned embeddings per axis**, shared by the attention layers of a video
  VQ-VAE and added once to the prior's inputs [arxiv:2104.10157 §3].
- **A sinusoidal code of time remaining** to a target. The same formula is used,
  but the input is a countdown, not a token index [arxiv:2102.04942 §3.3].
- **Learned box and point embeddings inside each decoder layer.** Each layer embeds
  the current box of every candidate and the coordinates of every point, and the
  boxes are refreshed from the previous stage [arxiv:2104.00678 §3.2].
- **A Gaussian sinusoidal encoding of an oriented box.** The box is treated as a
  2-D Gaussian, and its angle enters through the covariance [arxiv:2409.19648 §3.3].
- **A learned embedding for the exemplar tokens of a joint tracker.** It is
  computed from each patch's grid position and its overlap with the target box.
  The search tokens keep the pre-trained embedding [arxiv:2203.05328 §3.2]
  [eq. 7].
- **The integrated encoding of a conical frustum, with its bands fixed.** Detail
  at each zoom level comes from training stages, not from a change to the
  encoding [arxiv:2112.05504 §3.1] [arxiv:2112.05504 §3.3].
- **A frequency encoding of position, direction and time in a radiance field.**
  The encoding is applied to the inputs of each layer's MLP, and time enters it
  instead of a per-frame latent code [arxiv:2104.14786 §4.2].
- **A coarse-to-fine window on the same sine and cosine bands.** A radiance field
  starts with only its smooth bands open, and opens the rest during training, so
  camera poses can be fitted together with the scene [arxiv:2104.06405 §3.4].
- **Frequency bands on the input of a signed-distance and an albedo network.**
  Both networks encode the surface point with a frequency encoding, six bands
  for the shape and ten for the material [arxiv:2104.00674 §3].
- **A per-channel gate on the feature grid.** Nothing is added to the input. Each
  value is multiplied by a sigmoid of a depth-wise convolution over its neighbours
  [arxiv:2105.13677 §2.4] [arxiv:2105.13677 eq. 8].
- **No position encoding at all.** Overlapping convolutional token embeddings
  and depth-wise convolutional projections carry the local structure instead
  [arxiv:2103.15808 §3.3] [tab. 5].
- **A distance penalty added to attention scores.** Nothing is added to the
  embeddings. The penalty grows with the distance between query and key, with a
  fixed slope per head [arxiv:2211.05100 §3.2].
- **A convolutional encoder over the sequence, reshaped to a square grid.** The
  grid is summed with three convolutions of itself [arxiv:2106.00908 §3.3]
  [alg. 3].
- **Rotary embeddings on queries and keys**, in every layer, with no table of
  position vectors [arxiv:2302.13971 §2.2].
- **Rotary embeddings over global positions inside a chunked linear-time
  attention, plus a relative bias.** Each chunk is attended exactly, and the
  rotation is not restarted per chunk [arxiv:2202.10447 §3.2] [arxiv:2202.10447
  app. D].
- **A 2-D sine and cosine location code as a condition**, not added to the input.
  Each scale's normalising flow reads the code of its patch's row and column
  [arxiv:2107.12571 §4.2].
- **Position, date and band codes in a satellite model.** Each token gets a
  position code, a year, month and hour code, and a group code for its band
  group [arxiv:2207.08051 §4.1.1] [arxiv:2207.08051 §4.2].
- **A day-of-year code on each observation** in a per-pixel time series model
  [arxiv:2506.20380 §3.2].
- **A per-token camera-ray frame**, with a rotary encoding on the other half of
  the channels, and an optional latitude-up map for pitch and roll
  [arxiv:2512.07237 §3.2] [arxiv:2512.07237 §3.3].
- **A learned bucket table added to the attention scores.** Nothing is added to
  the patch embeddings. Each pair of patches is mapped to a bucket by its row and
  column offset, and each bucket holds a learned vector [arxiv:2107.14222 §3.2]
  [arxiv:2107.14222 eq. 15]. This is a relative encoding, and the wider comparison
  is on [relative-positional-encoding](/concepts/relative-positional-encoding).
- **An acyclic sinusoidal time code from a learned trajectory.** A one-dimensional
  convolution over sparse noise sets the wave parameters, so a video can be read
  at any real-valued time [arxiv:2112.14683 §3.1] [arxiv:2112.14683 eq. 3].
- **A learnable positional vector per node, updated in every layer.** It starts
  from random-walk return probabilities and is kept apart from the structural
  features, so a graph network can tell apart nodes whose neighbourhoods match
  [arxiv:2110.07875 §3.1].
- **A taxonomy of node positional bias for graph transformers.** Local options
  use hop distances, and global options use Laplacian eigenvectors or
  random-walk probabilities. It is a survey's classification, not a measured
  comparison [arxiv:2407.09777 §3.1.1].

Encodings that enter through the attention score itself, relative to the query
position, are covered on [relative-positional-encoding](/concepts/relative-positional-encoding),
which also discusses rotary embeddings as used by PaLM. The iRPE bucket table
and the StyleGAN-V time code have their own sections below. The rotary subsection
below covers the LLaMA and Code Llama cards. The RoPE2d that one card here uses
per scale [arxiv:2412.04431 §3.5] is described on neither page.

## Variants

### Fixed sinusoidal encoding added to the input

Each position gets a vector of size $d$. For each pair of entries $2i$ and
$2i+1$, the vector holds the sine and cosine of the position divided by
$10000^{2i/d}$ [arxiv:2205.01138 §2.5.1] [eq. 8] [eq. 9]. The encoding is summed
with the token embedding before the first encoder block [arxiv:2205.01138 §2.4.4].
It has no learnable parameters [arxiv:2205.01138 §2.5.1].

The tutorial argues that a fixed offset $k$ maps the encoding at position $i$ to
the encoding at $i+k$ by a rotation that depends only on $k$, so the encoding
carries relative position [arxiv:2205.01138 §2.5.3] [eq. 13]. That argument shows
the shift is a linear map. It does not show that a trained model uses it
[arxiv:2205.01138 §2.5.3]. The tutorial justifies summing rather than
concatenating by analogy, not proof [arxiv:2205.01138 §2.5.4].

The base is 10,000 in the text, which says it may change with sequence length
without giving a rule, but 1,000 in the illustration [arxiv:2205.01138 §2.5.1]
[fig. 8]. The tutorial's time-series variants keep this encoding and change the
input layer around it [arxiv:2205.01138 §3]. The tutorial runs no experiments,
so it gives no benchmark for this variant.

A second tutorial, on the Transformer for NLP, gives the same sine and cosine
formula with base 10000 and adds it to the word embedding [arxiv:2311.17633
§2.2]. It states that attention alone ignores word order, which is the reason for
the encoding [arxiv:2311.17633 §2.2]. Like the first, it runs no experiments
[arxiv:2311.17633 §1]. The two agree on the formula and the base, so they add no
variant of their own.

The satellite foundation model Prithvi-EO-2.0 uses the same sine and cosine form
in three dimensions. A 1D code is built for time, height and width, and the three
are combined into one 3D encoding and added to the patch tokens
[arxiv:2412.02732 §3.2] [arxiv:2412.02732 eq. 1]. Date and location metadata get
their own 1D sine and cosine codes, each with a learned weight, and are added as a
bias [arxiv:2412.02732 §3.2]. The paper gives the metadata drop probability as 0.1
for the TL models only [arxiv:2412.02732 §3.2]. The card reports no ablation of
the positional part.

The anomaly detector TranAD adds the same sinusoidal form to the window before
its first encoder [arxiv:2201.07284 §3.3]. Its ablations do not remove the
encoding, so no benefit is measured [arxiv:2201.07284 tab. 6]. The released code
adds both the sine and the cosine of one angle to every dimension, not one
angle per sine-cosine pair, so the released encoding differs from the cited one
[code: src/dlutils.py#L200-L206].

### Time-to-arrival code (sinusoid of frames remaining)

The motion in-betweening generator needs to know how far the target keyframe is.
It encodes the number of frames until that keyframe with the sinusoidal formula,
using dimension 256 and base 10,000 [arxiv:2102.04942 §3.2] [arxiv:2102.04942 §3.3].
The code is held constant beyond a maximum that depends on the transition and
past-context lengths [arxiv:2102.04942 §3.3]. The same code is added to all
three encoder outputs before the LSTM [arxiv:2102.04942 §1] [arxiv:2102.04942 §3.2].

On the Human3.6M walking benchmark, adding the code moves the cumulative
average L2Q from 0.97 to 0.88 and average L2P from 2.52 to 2.04. The next
step adds target noise and reaches 0.84 and 1.83 [arxiv:2102.04942 tab. 2].
These are author-reported, and each row adds one component to the row above.

### Learned axial embedding in a video VQ-VAE and prior (VideoGPT)

VideoGPT has no sinusoid. Its VQ-VAE uses learned spatio-temporal embeddings,
shared by all axial attention layers in the encoder and decoder [arxiv:2104.10157 §3].
The released code learns one table per axis, each of size embedding divided by
three, concatenates them and adds the result to the input
[code: videogpt/attention.py#L458-L493]. The paper says only "learned
spatio-temporal embeddings" [arxiv:2104.10157 §3]. The prior's training step
also adds a learned positional embedding to the shifted code embeddings
[code: videogpt/gpt.py#L139-L155]. The card reports no ablation of either
embedding, so no benefit is measured.

### Learned box and point embeddings per decoder layer (Group-Free 3D)

This model has no token order. Each candidate object attends to all seed points.
Self-attention among candidates receives a learned embedding of the candidate's
box, taken from the previous stage. Cross-attention to points receives a learned
embedding of each point's coordinates. Every layer has its own embeddings
[arxiv:2104.00678 §3.2].

The paper says the refined encoding is added to each stage's output and fed to
the next stage [arxiv:2104.00678 §3.2]. The released code adds it inside each
attention layer instead. See Contradictions.

On ScanNet V2, with 6 layers and 256 candidates, averaged over 25 trials, the
ablation reads as follows (mAP@0.25 / mAP@0.5) [arxiv:2104.00678 tab. 5]:

| Encoding | mAP@0.25 | mAP@0.5 |
|---|---|---|
| None | 64.7 | 43.4 |
| Fixed centre and size | 64.6 | 43.5 |
| Iterative, centre only | 65.2 | 47.5 |
| Iterative, centre and size | 66.3 | 48.5 |

Reading the table: a fixed encoding barely changes the result, and refreshing
the encoding from each stage's prediction accounts for most of the gain at
IoU 0.5 [arxiv:2104.00678 tab. 5]. The card says the paper gives no channel count,
head count or query initialisation for this block [arxiv:2104.00678 §3.2].

### Joint exemplar-search tokens with a learned exemplar embedding (SimTrack)

SimTrack puts the exemplar tokens and the search tokens in one sequence. A
separate learned embedding is added to the exemplar tokens only. It is computed
from each patch's grid position (i, j) and a fraction R_ij of the target area in
that patch, through two fully connected layers [arxiv:2203.05328 §3.2] [eq. 7].
The search tokens keep the pre-trained embedding, because the search image has
the same size as in standard ViT training [arxiv:2203.05328 §3.2]. The paper's
reason for a separate embedding is that the exemplar is smaller than the search
image, and one shared embedding would give the backbone no way to tell the two
images apart [arxiv:2203.05328 §3.2].

A foveal window adds 64×64 crops from the centre of the exemplar as extra
tokens, cut into patches with their own positions [arxiv:2203.05328 §3.3]
[app. Training Details]. The paper does not describe any output for the foveal
tokens, so they act as extra input only [arxiv:2203.05328 §3.3].

The paper's cumulative ablation on TNL2K, for ViT-B/16 with STARK-SV as the
base, reads as follows (AUC / precision) [arxiv:2203.05328 tab. 8]:

| Config | TNL2K AUC / Pre |
|---|---|
| Base (STARK-SV) | 52.2 / 51.1 |
| + joint framework | 53.5 / 52.5 |
| + distinguishable position embedding | 54.0 / 53.1 |
| + foveal window | 54.8 / 53.8 |

Each row adds one part to the row above, so the position embedding's +0.5 AUC
is measured on top of the joint framework only. Table 4 compares the whole
framework with STARK-SV on the same backbone. For ViT-B/16 the LaSOT AUC gain is
+2.5 [arxiv:2203.05328 tab. 4]. Results are author-reported, and the card has no
separate test of the embedding alone.

### Gaussian encoding of oriented boxes (OrientedFormer)

The detector predicts tilted rectangles, so the positional query is a box
(x, y, w, h, θ). The box is turned into a 2-D Gaussian with mean (x, y) and a
covariance that depends on the size and the angle [arxiv:2409.19648 §3.3]
[arxiv:2409.19648 eq. 2]. The sinusoidal encoding is then evaluated as the
expectation of a sine or cosine under that Gaussian. This equals the sine or
cosine of the mean, damped by a factor that depends on the variance
[arxiv:2409.19648 eq. 4] [arxiv:2409.19648 eq. 5]. Only the diagonal of the
covariance is needed [arxiv:2409.19648 eq. 6] [arxiv:2409.19648 eq. 7].

The same Gaussian geometry also biases decoder self-attention, through a
Wasserstein distance between boxes [arxiv:2409.19648 §3.4]. That bias is a
separate component, and the ablations below test it separately.

On DIOR-R, with ResNet-50 and AP50, the encoding ablation reads as follows
[arxiv:2409.19648 tab. 10]:

| Encoding | AP50 |
|---|---|
| None | 66.85 |
| Deformable | 65.97 |
| DAB | 66.03 |
| Learnable | 64.27 |
| Gaussian | 67.28 |

The module ablation on DIOR-R combines the Gaussian encoding with the Wasserstein
bias and with cross-attention [arxiv:2409.19648 tab. 11]. In that table, the
Gaussian encoding alone with cross-attention reaches 67.03, against 65.78 for
cross-attention alone. Both ablations come from one detector and one dataset.

### Frequency encoding of position, direction and time (ST-NeRF)

ST-NeRF has no token order. Each layer is a function of a 3D position $p$, a
view direction $d$ and a time $t$. A deformation MLP predicts an offset from
$(p, t)$, and a radiance MLP predicts colour and density at the moved point
[arxiv:2104.14786 §4.2] [eq. 2] [eq. 3]. The frame index enters through a
positional encoding, not through a per-frame latent code [arxiv:2104.14786 §4.2].
The paper does not state the number of frequencies. The released code uses
L = 10 for position and for the deformation input, and L = 4 for direction
[code: modeling/spacenet.py#L20-L22] [code: modeling/motion_net.py#L14]. The card
reports no ablation of this encoding, so no benefit is measured.

### Coarse-to-fine window on the frequency bands (BARF)

BARF uses the NeRF encoding of the previous variant: the raw coordinates, plus
a sine and a cosine at each frequency $2^k \pi$ [arxiv:2104.06405 eq. 10]
[arxiv:2104.06405 eq. 11]. Its window multiplies only the sine and cosine of
band $k$. The window is zero while training progress $\alpha$ is below $k$, rises
along a half-cosine over one unit of progress, and is one once $\alpha$ reaches
$k+1$ [arxiv:2104.06405 eq. 13] [arxiv:2104.06405 eq. 14]. The raw coordinates
are never masked [arxiv:2104.06405 §3.4]. Progress grows linearly from iteration
20K to 100K of a 200K run, and all bands stay open after that [arxiv:2104.06405
§4.2] [arxiv:2104.06405 §4.3].

The paper's reason for the window is that the derivative of a band scales with
its frequency, so the high bands make the pose gradients hard to follow
[arxiv:2104.06405 §3.3] [arxiv:2104.06405 eq. 12]. The window also closes the
gradient of a masked band, so an open band has no effect on the poses until
it opens [arxiv:2104.06405 eq. 15].

The schedule is hand-set. The paper uses the same window for all LLFF scenes
and calls a per-scene schedule out of scope [arxiv:2104.06405 app. B.2]. The
released code treats the window as a fraction of the total iterations, so a
shorter run opens the bands earlier [code: model/barf.py#L57].

On the eight Blender scenes with perturbed poses, mean rotation error falls
from 6.167 to 0.193 degrees against full encoding, and PSNR rises from 22.12 to
27.50 dB [arxiv:2104.06405 tab. 2]. A NeRF with no positional encoding reaches
0.202 degrees and 26.78 dB on the same scenes. Against that baseline, the window
adds little on rotation, and its PSNR gain is 0.72 dB [arxiv:2104.06405 tab. 2]. On the eight LLFF
scenes from identity poses, BARF's rotation error is 0.573 degrees against
0.699 for the no-encoding baseline, but its translation error is 0.331 against
0.298, and its PSNR is 23.97 against 23.82 dB [arxiv:2104.06405 tab. 3]
[arxiv:2104.06405 tab. 5]. The paper calls it competitive with that baseline
rather than uniformly better. With metric depth in place of inverse depth,
BARF's LLFF rotation error rises to 1.580 degrees, and the baseline is better
on rotation, translation and PSNR [arxiv:2104.06405 tab. 6].

### Frequency bands on the surface point of an SDF and an albedo network (PhySG)

PhySG's geometry network is a signed distance function, and its albedo network
gives the diffuse colour at a surface point. Both use a frequency encoding of
that point. The shape network uses six frequencies and the albedo network ten
[arxiv:2104.00674 §3] [code: code/confs_sg/default.conf#L43]
[code: code/confs_sg/default.conf#L47-L48]. The paper gives the frequency form
in a footnote [arxiv:2104.00674 §3].

The paper's results come from the full pipeline, which also learns the lighting
and the material. They do not isolate this encoding [arxiv:2104.00674 tab. 1].
The card gives no ablation of it.

### Integrated encoding with scale-staged training (BungeeNeRF)

BungeeNeRF uses the integrated positional encoding of Mip-NeRF, and does not
change it. Each sampled interval is a conical frustum, and its encoding is the
sine and cosine of the frustum's mean at frequencies 2^m for m = 0 to 9, each
damped by a factor that depends on the frustum's variance [arxiv:2112.05504 §3.1]
[arxiv:2112.05504 §4]. The encoding is a fixed function of the frustum, and the
scale of an image is handled by training. Training runs in stages, starting from
the most distant scale. Each stage adds the next closer scale and one residual
block with its own output, and each output is trained on all scales up to its
own [arxiv:2112.05504 §3.3] [eq. 3].

The motivation is that close views need high-frequency bands and far views do
not. Training one NeRF on all scales at once biases it toward low frequencies,
so the paper argues close views blur [arxiv:2112.05504 §3.2] [fig. 2]. Because
the encoding stays fixed, the card cannot attribute any gain to it.

The ablations remove the training scheme, not the encoding. On 56 Leonard, the
average PSNR is 23.481 for the full model, 22.139 without model growing and
22.340 without the data feeding scheme [arxiv:2112.05504 tab. 3]. Results are
author-reported. The "same iter as baselines" row uses the baselines' budget,
and the "until convergence" row trains longer, so the two rows are not
like-for-like [arxiv:2112.05504 tab. 2]. The main baseline, Mip-NeRF, uses the
same integrated encoding, so that comparison does not test the encoding either.

The released code keeps the raw mean as a first input feature, which the
paper's formula omits. It gives 63 input dimensions for 10 bands
[code: run_nerf_helpers.py#L59-L61] [code: run_nerf_helpers.py#L126-L142]. Use
that size if you follow the code rather than the formula.

### Depth-wise convolution gate, pixel attention (ResT)

ResT has no additive vector and no table tied to the input size. Each value is
multiplied by a sigmoid of a 3x3 depth-wise convolution of the same grid
[arxiv:2105.13677 §2.4] [arxiv:2105.13677 eq. 8]. The paper says this lets the
network take any input size [arxiv:2105.13677 §2.4]. It compares this gate with a
learned absolute table (LE) and a per-channel linear map (GL) [arxiv:2105.13677 §2.4]
[arxiv:2105.13677 app. A.4].

On ResT-Lite at 224 px, with the simple training recipe, the top-1 results read as
follows [arxiv:2105.13677 tab. 8]:

| Encoding | Top-1 | Top-5 |
|---|---|---|
| None | 71.54 | 89.82 |
| LE (learned absolute) | 71.98 | 90.32 |
| GL (group linear) | 72.04 | 90.41 |
| PA (pixel attention) | 72.88 | 90.62 |

The paper argues that PA encodes position and allows any input size
[arxiv:2105.13677 §2.4]. The ablation compares only ResT-Lite at 224 px, so it
does not test different input sizes [arxiv:2105.13677 tab. 8]. The reference code
computes height and width from the input, with no learned table tied to the input
size [code: models/rest.py#L164-L172]. The code's GL is not the paper's GL: the
code's version is a residual depth-wise convolution, and no model in the repository
calls it [code: models/rest.py#L142-L148].

### No position embedding, convolutions instead (CvT)

CvT drops the position embedding from all three stages. Its tokens come from an
overlapping convolution, and each attention block computes queries, keys and
values with depth-wise convolutions [arxiv:2103.15808 §3.1] [arxiv:2103.15808 §3.2].
The authors say local relations are modelled by these convolutions, which also
helps with inputs of varying resolution [arxiv:2103.15808 §3.3] [arxiv:2103.15808 §4.4].

On ImageNet-1k top-1, the position-embedding ablation reads as follows
[arxiv:2103.15808 tab. 5]:

| Setting | Top-1 |
|---|---|
| DeiT-S, default position embedding | 79.8 |
| DeiT-S, no position embedding | 78.0 |
| CvT-13, position embedding in every stage | 81.5 |
| CvT-13, position embedding in first stage only | 81.4 |
| CvT-13, position embedding in last stage only | 81.4 |
| CvT-13, no position embedding (default) | 81.6 |

Table 6 removes the convolutional token embedding and crosses it with the
position embedding. Without the convolutional embedding, top-1 is 80.7 with no
position embedding and 81.1 with one. With it, the values are 81.4 with a
position embedding and 81.6 without [arxiv:2103.15808 tab. 6]. So within CvT-13
the position embedding moves top-1 by 0.2 points or less. The paper's variable
resolution claim is not tested on ImageNet [arxiv:2103.15808 §4.4]. The
ablation is for the classification setting only.

### Convolutional encoder over a square grid (PPEG, TransMIL)

Attention over a bag of patches has no grid, so TransMIL pads the sequence to a
square, reshapes the patch tokens into a grid in input order, and sums the grid
with three convolutions of it, with kernels 7, 5 and 3 [arxiv:2106.00908 alg. 3]
[arxiv:2106.00908 §3.3]. The convolutions are grouped per channel
[code: models/TransMIL.py#L32-L34]. The model has two attention layers, and this
step sits between them [arxiv:2106.00908 alg. 2]. "Spatial" here means sequence
order, not (x, y) coordinates, so the spatial signal depends on how the feature
files were written [code: README.md#L33].

On CAMELYON16 and NSCLC, scored by AUC, the position-encoding ablation reads as
follows [arxiv:2106.00908 tab. 2]:

| Position encoding | Params | CAMELYON16 AUC | NSCLC AUC |
|---|---|---|---|
| none | 2.625M | 0.8416 | 0.9287 |
| sinusoidal (scaled by 0.001) | 2.625M | 0.8941 | 0.9374 |
| 3x3 conv | 2.630M | 0.8913 | 0.9355 |
| 7x7 conv | 2.651M | 0.9015 | 0.9336 |
| both | 2.669M | 0.9059 | 0.9402 |
| PPEG | 2.669M | 0.9309 | 0.9603 |

Every row with a position encoding beats "none", and PPEG is highest on both
datasets [arxiv:2106.00908 tab. 2]. The paper reports no standard deviations
[§4]. The "both" row is not defined in the text [arxiv:2106.00908 tab. 2]. The
card cannot check the ablation, because its scripts are not in the repository
[code: README.md#L33].

### ALiBi attention bias (BLOOM)

BLOOM adds no position vector to its embeddings. For each head, it adds a term
to the attention score that falls with the distance between key and query. The
slope is fixed for each head [arxiv:2211.05100 §3.2]. The figure caption gives
the slopes as $2^{-8i/n}$ for head $i$ among $n$ heads [fig. caption §3.2].

The authors chose this over learned or rotary embeddings, based on ablations
run at 1.3B parameters [arxiv:2211.05100 §3.2]. The card gives no numbers for
that comparison. BLOOM has
112 heads, which is not a power of two. The reference code uses a different slope
rule in that case, so the caption formula is not what the released model
computed [code: megatron/model/transformer.py#L596-L607]. The paper does not say
which rule the trained model used. The model was trained on sequences of 2048
tokens [arxiv:2211.05100 §3.5], and the card cites no test beyond that length.

### Rotary embeddings on queries and keys (LLaMA, Code Llama)

LLaMA rotates the query and key vectors of every attention layer by a
position-dependent angle, and uses no absolute position embedding
[arxiv:2302.13971 §2.2]. The paper does not state the rotation base or the
context length. The released code uses a base of 10000 and rotates adjacent
channel pairs, not the two halves of each head [code: llama/model.py#L80]
[code: llama/model.py#L156-L161]. Other codebases split the halves, which
changes the attention scores relative to the code. The LLaMA card reports no
ablation of the encoding.

Code Llama keeps the rotary form and raises the base from 10^4 to 10^6 for
long-context fine-tuning on 16,384-token sequences, with a learning rate of
2e-5 [arxiv:2308.12950 §2.4]. Frequencies follow base^(-2i/d), one per pair of
dimensions [code: llama/model.py#L55-L56]. The paper says the change is
different from position interpolation, because the period of the rotation is
changed instead of scaling the frequencies linearly [arxiv:2308.12950 §2.4].
The paper reports that the change can be applied to a pretrained model, and
that the loss settles within a few steps at a low learning rate [arxiv:2308.12950
app. G].

Fine-tuning with the raised base improves single-line completion on LCC-balanced
at 7B, from 36.86 to 39.23 exact match. It lowers HumanEval pass@1 by 0.52 points
and MBPP pass@1 by 1.9 points on average [arxiv:2308.12950 §3.3]. These results
are for code tasks only. RoPE itself has no card here; see
[relative-positional-encoding](/concepts/relative-positional-encoding) for PaLM's
use of it.

### Rotary positions across chunks, with a relative bias (FLASH)

FLASH cuts the sequence into chunks. Inside each chunk it computes exact
squared-ReLU attention, and between chunks it uses linear attention
[arxiv:2202.10447 §3.2]. The attention score carries a relative position bias
b, which the paper calls a relative position bias but does not define
[arxiv:2202.10447 §2 eq. 5]. The appendix listing applies rotary embeddings to
queries and keys over the whole sequence, with the rotation not restarted in
each chunk, so every token keeps its global position [arxiv:2202.10447 app. D].
The same listing builds b in one of two ways, chosen by the length argument: a
learned table of 2n−1 weights below length 512, or a Toeplitz matrix from
rotated vectors at 512 and above [arxiv:2202.10447 app. D]. The paper's
hyperparameter tables list RoPE on queries and keys [arxiv:2202.10447 tab. 8].

The position scheme is not measured on its own. The main comparison pits FLASH
against Transformer++, which also uses rotary embeddings and a gated linear
unit, while FLASH also changes the attention, the gate and the chunking
[arxiv:2202.10447 §4]. At 8192 tokens on Wiki-40B, FLASH reaches perplexity
15.109 against 15.254 for Transformer++, at 62.5 against 261.0 ms per training
step on 64 TPU-v4 cores [arxiv:2202.10447 tab. 13]. Those are author-reported
results.

### Sinusoidal location code as a flow condition (CFLOW-AD)

CFLOW-AD adds nothing to its features. Each scale has its own conditional
normalising flow, and that flow is conditioned on a fixed 2-D code of each
location, with 128 channels in all experiments [arxiv:2107.12571 §4.2]. The
code is built from sine and cosine harmonics, and half of the channels encode
the row and half the column [code: model.py#L11-L30]. The condition is
concatenated inside each coupling layer [arxiv:2107.12571 §4.2].

The paper gives no ablation of the code by itself. Its one flow comparison
replaces the conditional flow with an unconditional one, which removes the
condition as a whole. On MVTec AD with WRN50, 8 blocks and 3 scales at 512 px,
the average localisation AUROC is 97.36 for the conditional flow and 96.86 for
the unconditional one [arxiv:2107.12571 tab. 2]. The paper reports the
conditional flow as about 0.5 points better on average [arxiv:2107.12571 §5.2].

### Year, month and hour codes with band-group codes (SatMAE)

SatMAE's temporal path gives each token two codes that are concatenated to the
embedding width. The first is a positional code for the token's place in the
image. The second is built from the year, month and hour of the token's image,
and drops the day, minute and second [arxiv:2207.08051 §4.1.1]
[arxiv:2207.08051 eq. 2]. In the temporal runs the positional code has 640
dimensions and the temporal code 384, split across the three fields
[arxiv:2207.08051 app. A.3.2]. The paper does not give an origin for the year,
so an implementation must choose one and keep it the same in pre-training and
fine-tuning.

For multi-band input, the bands form groups, each with its own patch
embedding. A sinusoidal code of the group index is concatenated to the
positional code [arxiv:2207.08051 eq. 3] [arxiv:2207.08051 §4.2]. For the
Sentinel-2 runs the positional code has 768 dimensions and the group code 256
[arxiv:2207.08051 app. A.3.3].

On fMoW-Sentinel validation, top-1 is 59.30 with the group code and 58.87
without it, with the same grouping and independent masking
[arxiv:2207.08051 §5.4]. This is the cleanest isolated test of a code in the
card. The temporal ablation is not as clean: the row without the temporal
encoding (78.07) uses independent masking, and the row with it (78.45) uses
consistent masking, so the encoding's own effect is not measured
[arxiv:2207.08051 §5.3].

### Day-of-year encoding of observation dates (TESSERA)

TESSERA's encoder works on a set of valid observations for one pixel. Each
sensor's encoder adds a day-of-year (DOY) encoding to each date's embedding
[arxiv:2506.20380 §3.2]. The main text describes this encoding as learnable.
The appendix describes sine and cosine of the normalised DOY [arxiv:2506.20380
§3.2] [app. Pre-training Details]. The ablation removes global shuffling,
mixup, Sentinel-1, quantisation and pre-training, but not the DOY encoding
[arxiv:2506.20380 tab. 3]. So the card has no measured effect for it.

### Ray-frame encoding with a latitude-up map (UCPE)

UCPE gives each token one viewing ray, with the ray's origin and direction
[arxiv:2512.07237 eq. 5]. A local frame is built for each ray, with the ray
direction as the z-axis, the cross product of the camera's downward axis and z
as the x-axis, and y completing the frame [arxiv:2512.07237 eq. 6]. Attention
uses the inverse transform [arxiv:2512.07237 eq. 7]. The ray transform takes
half of the feature dimensions and RoPE takes the other half, as one
block-diagonal operator per token [arxiv:2512.07237 eq. 10]
[arxiv:2512.07237 §3.3]. Lens effects are carried by the rays, not by a
separate intrinsic matrix [arxiv:2512.07237 §3.2].

The latitude-up map is optional. It gives each pixel a latitude and a two-part
up vector, which a linear layer projects to the token width and adds as a bias
[arxiv:2512.07237 §3.2] [arxiv:2512.07237 §3.3]. The paper uses a rotation of
0.1 rad to build the up vector [arxiv:2512.07237 app. B].

The ray encoding is compared with two replacements inside the same adapter.
With PRoPE, rotation error is 5.35 degrees and camera-motion error 20.58; with
GTA, 5.27 and 20.14. The UCPE default gives 4.12 and 17.59 on the same
UCM-trained setting [arxiv:2512.07237 tab. 1]. The latitude-up map is reported
with its pitch and roll numbers only. Without it, rotation error is 4.29, but
translation error (13.46 against 15.21) and camera-motion error (15.94 against
17.59) are lower, so the map does not improve those two metrics
[arxiv:2512.07237 tab. 1]. The training data is synthetic, and the zero-shot
RealEstate10K test uses 100 clips [arxiv:2512.07237 app. A]
[arxiv:2512.07237 tab. 2].

### Learnable positional stream updated in every layer (LSPE, graph networks)

Message-passing graph networks update a node from its neighbours, so two nodes
with the same neighbourhood stay alike [arxiv:2110.07875 §1]. The LSPE card adds
a second per-node stream of position values. The structural update reads the
concatenation of structural and positional features. The positional update is
a separate message-passing function, and a tanh lets the coordinates take either
sign [arxiv:2110.07875 §3.1] [eq. 7] [eq. 8] [eq. 9]. The comparison point is a
baseline that concatenates a fixed encoding to the input and never updates it
[arxiv:2110.07875 §3.1] [eq. 3].

The initial vector is either random-walk return probabilities (RWPE) or
Laplacian eigenvectors (LapPE). RWPE has no sign ambiguity. LapPE needs random
sign flips during training [arxiv:2110.07875 §3.1]. The RWPE is precomputed
before training [arxiv:2110.07875 app. A].

On ZINC, GatedGCN with RWPE and the per-layer stream reaches test MAE
0.093 ± 0.003. Without any encoding it reaches 0.251 ± 0.009, and with LapPE
0.202 ± 0.006. The Laplacian auxiliary loss lowers the first value to
0.090 ± 0.001 [arxiv:2110.07875 tab. 1]. Injecting RWPE at the input without the
per-layer stream gives 0.122 ± 0.003 [arxiv:2110.07875 tab. 3]. Most of the
gain in the card therefore needs the per-layer update, not the RWPE alone. The
numbers are author-reported, averaged over 4 seeds [arxiv:2110.07875 §4.1].

On the two OGB molecular sets the changes are small. GatedGCN with RWPE reaches
0.775 ± 0.003 test ROC-AUC on MOLTOX21, against 0.772 ± 0.006 without an
encoding. The two runs' spreads overlap [arxiv:2110.07875 tab. 1]. The card
reads these as comparable, not as wins [arxiv:2110.07875 §4.2].

### Node positional bias in graph transformers (survey taxonomy)

A graph transformer can add a vector to each node that says where the node
sits in the graph. The survey sorts these into local and global options
[arxiv:2407.09777 §3.1.1]. Local options are one-hot hop distances, learned
embeddings of edge types, and graph-kernel similarities between
neighbourhoods. Global options are Laplacian eigenvectors or eigenvalues,
random-walk or diffusion probabilities such as personalised PageRank and the
heat kernel, and a low-dimensional embedding that keeps graph distance
[arxiv:2407.09777 §3.1.1].

Edge structure enters attention in a related way, as a bias on the scores or as
extra tokens. The survey's examples include a shortest-path kernel and global
spectral encodings [arxiv:2407.09777 §3.1.2]. It also places a message-passing
step before, between or after the attention layers [arxiv:2407.09777 §3.1.3].

The survey is a classification, and its tables give no performance numbers
[arxiv:2407.09777 §4]. Its only numeric guidance is a design guide, which lists
adding structural and positional encodings as one principle [arxiv:2407.09777 §6].
Use the original papers for any measured effect.

### Renumbering positions after cache compaction (HERMES)

This is not an encoding. A streaming video model that drops cached tokens keeps
the positions of the survivors in range. HERMES renumbers them when positions
approach the model's maximum, and corrects the cached keys with a rotary delta
instead of recomputing them [arxiv:2601.14724 §3.3] [app. E]. Lazy renumbering
runs only near the limit, and eager renumbering runs at every compression step
[arxiv:2601.14724 §4.1] [arxiv:2601.14724 §4.4]. The paper does not state the
margin before the limit. The reference code starts compaction 1024 positions
early [code: inference/llavaov_hermes.py#L171-L179].

The ablation compares lazy and eager renumbering with the base model. On
StreamingBench real-time and OVO-Bench, the OVO average is 56.94 for lazy and
56.06 for eager, against 53.35 for the base model [arxiv:2601.14724 tab. 8]. On
VideoMME, the average is 54.93 for lazy and 58.44 for eager, against 57.67 for
the base model [arxiv:2601.14724 tab. 9]. The table has no row that compacts the
cache without renumbering, so the renumbering effect is not isolated.

### Learned bucket table in the attention scores (iRPE)

iRPE adds a learned term $b_{ij}$ to each attention logit before the division
by $\sqrt{d}$ [arxiv:2107.14222 eq. 13]. The bucket of a pair of patches comes
from a piecewise index on its row offset and on its column offset. The index is
exact up to $\alpha$, log-spaced up to $\gamma$, and capped at $\beta$
[arxiv:2107.14222 eq. 18]. The Product mapping gives one bucket per (row
bucket, column bucket) pair, and the paper's main setting is contextual mode,
where each bucket vector is dotted with the query and the table is shared across
heads [arxiv:2107.14222 §3.2] [tab. 2]. The paper sets the table to zero at the
start [arxiv:2107.14222 supp. Weight Initialization].

On DeiT-S, ImageNet top-1, the rows that matter read as follows
[arxiv:2107.14222 tab. 4]:

| Setting | Top-1 |
|---|---|
| Learnable absolute encoding, no relative term | 79.9 |
| No absolute encoding, relative term on keys | 80.9 |
| No absolute encoding, relative term on queries, keys and values | 81.3 |
| Learnable absolute encoding, relative term on queries, keys and values | 81.4 |

The paper reads these as relative encodings replacing the absolute one for
classification. Detection is different. On COCO with DETR at 150 epochs, the
sinusoidal encoding alone gives 39.5 AP, with no encoding 30.4, and the
contextual 9×9 relative table alone 38.7. With both it gives 40.8
[arxiv:2107.14222 tab. 6 rows 1, 2, 4, 6]. The authors also report at most 1%
extra MACs for DeiT-S across the resolutions they test [arxiv:2107.14222 §4.2]
[fig. 4]. All numbers are author-reported.

### Acyclic time embedding for a continuous-time video generator (StyleGAN-V)

StyleGAN-V has no token index. Its input is a time, and a clip can be read at
any real-valued time. The authors argue that a standard sine embedding is
periodic, so a generated video repeats itself [arxiv:2112.14683 §3.1]. The
motion noise sits on a grid of times spaced $\delta^z$ apart, with 16 frames by
default and 256 for SkyTimelapse. Two padding-free one-dimensional convolutions
with kernel 11 and hidden size 512 turn that noise into a trajectory
[arxiv:2112.14683 app. B]. For a query time, the grid code on its left sets the
amplitudes, phases and periods of the waves. A stitching term makes the
embedding start and end at the grid points, and alignment vectors restore the
expressiveness that stitching removes [arxiv:2112.14683 §3.1]
[arxiv:2112.14683 eq. 4]. The resulting 512-dimensional code is tiled into the
constant input of the StyleGAN2 synthesis network [arxiv:2112.14683 fig. 5].

The paper does not test the sinusoidal form by itself. Its motion ablation swaps
the convolutional trajectory for LSTM codes. On FaceForensics, FVD16 is 47.41 by
default, 180.3 with LSTM codes at $\delta^z = 16$, and 131.9 at $\delta^z = 1$
[arxiv:2112.14683 tab. 2]. That changes the whole motion generator, so it does
not isolate the embedding. The authors list remaining periodic motion as a
limit [arxiv:2112.14683 app. A].

The discriminator also receives the gap between the frames of a clip as a time
condition. Removing it raises FaceForensics FVD16 from 47.41 to 95.4, and
SkyTimelapse FVD16 from 79.52 to 102.1 [arxiv:2112.14683 tab. 2]. The form of
that encoding differs between paper and code (see Contradictions).

## When to use which

Most cards give ablations for only some of these variants, and each ablation is
measured inside its own system. The strongest comparisons are the time-to-arrival
code, the iterative box encoding, the Gaussian oriented-box encoding, the ResT
gate and the PPEG table. Choose by the constraint, then check the card before trusting a benefit.

| Situation | Use | Why |
|---|---|---|
| Standard Transformer over a token sequence, with no position parameters to learn | Fixed sinusoidal encoding added to the input | It has no learnable parameters, and its offset property is stated in the tutorial [arxiv:2205.01138 §2.5.1] [arxiv:2205.01138 §2.5.3]. A second tutorial gives the same formula [arxiv:2311.17633 §2.2]. No card compares it with another scheme on a benchmark. |
| Space-time grid of latent codes or patches, one embedding per axis | Learned per-axis embedding, or the fixed 3D sinusoid | VideoGPT learns one table per axis [code: videogpt/attention.py#L458-L493]. Prithvi-EO-2.0 uses the sinusoid in 3D [arxiv:2412.02732 §3.2]. Neither card ablates its encoding. |
| Radiance field fitted to images whose camera poses are unknown or imperfect | Coarse-to-fine window on the frequency bands | On synthetic scenes the window helps little beyond removing the encoding: rotation error 0.193 against 0.202 with no encoding [arxiv:2104.06405 tab. 2]. On LLFF translation the no-encoding baseline is better [arxiv:2104.06405 tab. 3] [arxiv:2104.06405 tab. 5]. The schedule is hand-set [arxiv:2104.06405 app. B.2]. |
| Neural surface and material fitted to posed images of one object under static light, where the input is a 3D point | Frequency bands on the point, six for the shape network and ten for the albedo network | The band counts are stated in the paper [arxiv:2104.00674 §3]. Results come from the full pipeline, so the encoding's own effect is not measured [arxiv:2104.00674 tab. 1]. |
| Vision backbone that should accept any input size without a position table | Depth-wise convolution gate (pixel attention) | The gate lifts top-1 from 71.54 to 72.88 over no encoding on ResT-Lite at 224 px [arxiv:2105.13677 tab. 8]. Tested at one resolution only. |
| Decoder-only language model, where a distance penalty on scores is preferred to an embedding table | ALiBi | The authors picked it over learned and rotary embeddings in their small-scale ablations, and BLOOM used it at 2048 tokens [arxiv:2211.05100 §3.2] [arxiv:2211.05100 §3.5]. No numbers in the card. |
| Attention over a set of objects or points, with no sequence order | Learned coordinate embeddings per layer, refreshed each stage | The iteration ablation gives the gain at IoU 0.5 [arxiv:2104.00678 tab. 5]. Measured on indoor 3D detection only. |
| Graph regression on molecules, where nodes with the same neighbourhood must be told apart and a second node stream is affordable | Learnable positional stream updated per layer, initialised from RWPE | ZINC test MAE falls to 0.093 with the stream, against 0.122 with RWPE at the input only and 0.251 with no encoding [arxiv:2110.07875 tab. 1] [arxiv:2110.07875 tab. 3]. It adds about 18k parameters to GatedGCN on ZINC, 504309 to 522870 [arxiv:2110.07875 tab. 1]. The OGB molecular gains are within the spreads [arxiv:2110.07875 tab. 1]. |
| Graph transformer whose nodes must carry their place in the graph | Node positional bias from the survey taxonomy: hop distances for local attention, Laplacian eigenvectors or random-walk probabilities for global | The survey lists these options and recommends positional encodings, but gives no measured comparison [arxiv:2407.09777 §3.1.1] [arxiv:2407.09777 §6]. For measured gains from random-walk and Laplacian encodings in message-passing networks, see [arxiv:2110.07875 tab. 1]. |
| Detecting rotated boxes with a DETR-style decoder | Gaussian encoding of the oriented box | Beats none, deformable, DAB and learnable encodings on DIOR-R [arxiv:2409.19648 tab. 10]. Measured with one detector on remote-sensing data. |
| Continuous function of space, direction and time, where time is an input rather than a token index | Frequency encoding of the inputs | Used per layer in ST-NeRF, with L = 10 for position and time in the code [arxiv:2104.14786 §4.2] [code: modeling/spacenet.py#L20-L22]. No ablation in the card. |
| A cache-based model drops tokens and must keep positions in range | Renumber the surviving positions and correct cached keys with a rotary delta | Lazy and eager renumbering both beat the base model in HERMES's tables, but the table does not isolate the renumbering effect [arxiv:2601.14724 §3.3] [arxiv:2601.14724 tab. 8] [arxiv:2601.14724 tab. 9]. |
| The model must know how long remains until a target | Sinusoidal code of frames remaining | Adding the code lowers the average L2Q and L2P in the cumulative ablation [arxiv:2102.04942 tab. 2]. Measured on motion in-betweening only. |
| Time-series forecasting or classification | Fixed sinusoidal encoding, or a variant the paper names | The tutorial lists variants and a traffic example with 288 daily and 7 weekly positions, but gives no outcome for the encoding [arxiv:2205.01138 §4.4]. |
| Bag of many instances with an order, such as the patches of a slide, where a local convolution over the order is wanted | Convolutional encoder over a square grid (PPEG) | It beats no encoding, sinusoids, and single 3x3 or 7x7 convolutions on CAMELYON16 and NSCLC AUC [arxiv:2106.00908 tab. 2]. Two datasets, one model, and no standard deviations. |
| Vision backbone where local structure should come from convolutions, so no position table is needed | No position encoding, with a convolutional token embedding and depth-wise projections | CvT-13 without a position embedding reaches 81.6 top-1 on ImageNet-1k, against 81.5 with one in every stage [arxiv:2103.15808 tab. 5]. Tested on ImageNet only, so the resolution claim is unmeasured [arxiv:2103.15808 §4.4]. |
| Decoder-only LLM whose context must grow after pretraining | Rotary embeddings, with the base raised and a short fine-tune on longer sequences | Code Llama's base change gave 36.86 to 39.23 exact match on single-line completion at 7B, but cost 0.52 HumanEval pass@1 on average [arxiv:2308.12950 §2.4] [arxiv:2308.12950 §3.3]. Measured on code tasks only. |
| Long-context language model trained with chunked linear-time attention, where the positions must stay global across chunks | Rotary embeddings on queries and keys over global positions, plus a relative bias in the chunk | FLASH reports 15.109 perplexity against 15.254 for Transformer++ at 8192 tokens, with 62.5 against 261.0 ms per step [arxiv:2202.10447 tab. 13]. The baseline also uses rotary embeddings, so these results do not isolate the position scheme [arxiv:2202.10447 §4]. The bias is not defined in the paper [arxiv:2202.10447 §2 eq. 5]. |
| Anomaly scoring with a density model over a feature grid, where the model should know each location | Sinusoidal location code as the flow condition | CFLOW-AD reports 98.26 image AUROC on MVTec AD, author-reported [arxiv:2107.12571 tab. 4]. The only ablation removes the whole condition, so the code's own effect is not measured [arxiv:2107.12571 tab. 2]. |
| Masked pretraining on image sequences or many spectral bands | Year, month and hour code, plus a group code per band group | The group code gives 59.30 against 58.87 top-1 on fMoW-Sentinel validation [arxiv:2207.08051 §5.4]. The temporal encoding is confounded with masking in the card's ablation [arxiv:2207.08051 §5.3]. |
| Per-pixel model over irregular dated observations | Day-of-year encoding on each observation | The encoding is not in the ablation [arxiv:2506.20380 tab. 3]. The paper describes its form two ways, so pick one and test it [arxiv:2506.20380 §3.2]. |
| Vision transformer on a fixed patch grid, where attention should see row and column offsets between patches | Learned bucket table in the attention scores, Product mapping | On DeiT-S, relative terms on queries, keys and values reach 81.4 top-1 with the absolute encoding kept, against 79.9 [arxiv:2107.14222 tab. 4]. On DETR the absolute encoding still matters: relative alone gives 38.7 AP, against 39.5 with the sinusoid [arxiv:2107.14222 tab. 6]. Author-reported on ImageNet and COCO. |
| Joint tracker that must compare a fixed exemplar with each new frame, where the exemplar needs to be told apart from the search image | Learned exemplar embedding from patch grid position and target overlap, with the search tokens keeping the pre-trained embedding | The cumulative ablation gives +0.5 AUC on TNL2K for the embedding, on top of the joint framework [arxiv:2203.05328 tab. 8]. The exemplar's features depend on each frame, so they run with every frame [arxiv:2203.05328 §3.2] [code: lib/test/tracker/simtrack.py#L65]. Author-reported, with one backbone family. |
| Multi-scale radiance field with camera distances spanning several zoom levels, where one model must give each level | Integrated encoding kept fixed, with training stages that add scales and residual blocks | On 56 Leonard, the average PSNR is 23.481 for the full model against 22.312 for Mip-NeRF-full at the baselines' budget [arxiv:2112.05504 tab. 2] [arxiv:2112.05504 tab. 3]. The card cannot separate the encoding from the staged training [arxiv:2112.05504 §4.2]. |
| Video generator that must produce frames at any real-valued time, and whose clips should not repeat | Acyclic time embedding from a learned convolutional trajectory | FaceForensics FVD16 is 47.41 by default, against 180.3 with LSTM motion codes at $\delta^z = 16$ [arxiv:2112.14683 tab. 2]. The embedding alone is not ablated, and periodic motion still appears sometimes [arxiv:2112.14683 app. A]. |
| Camera-controlled video generation, with lenses that are not pinhole, or with pitch and roll to set | Ray-frame encoding, with the latitude-up map if pitch and roll matter | Rotation error is 4.12 degrees with the ray encoding, against 5.27 and 5.35 when GTA or PRoPE replaces it [arxiv:2512.07237 tab. 1]. Translation and camera-motion error are lower without the latitude-up map in the same table [arxiv:2512.07237 tab. 1]. Trained on synthetic clips only [arxiv:2512.07237 app. A]. |

## Implementation sketch

The shared core is the fixed sinusoid. The other variants change what goes in
(a countdown, a box, a learned table), where it goes (per layer, or into the
score), or both.

```pseudocode
# Fixed sinusoidal encoding, added to the input embedding  [arxiv:2205.01138 §2.5.1]
# Output: (n, d) with one row per position
def sinusoidal_encoding(n, d, base=10000):
    enc = zeros(n, d)
    for pos in range(n):
        for i in range(d // 2):
            # One frequency per sine/cosine pair  [eq. 8] [eq. 9]
            angle = pos / base ** (2 * i / d)
            enc[pos, 2 * i] = sin(angle)
            enc[pos, 2 * i + 1] = cos(angle)
    return enc
# x = token_embedding + sinusoidal_encoding(n, d)  [arxiv:2205.01138 §2.4.4]


# Time-to-arrival code: the same function applied to a countdown  [arxiv:2102.04942 §3.3]
# pos = min(frames_left, cap); held constant past the cap, cap value derived in card
# code = sinusoidal_encoding(1, 256)[0] with pos replaced by that value


# Learned per-axis embedding, concatenated across axes  [code: videogpt/attention.py#L458-L493]
# One table per axis, each of size d / 3, then concatenated and added to the input
def axial_embedding(t, h, w, d):
    emb_t = learned_table(t, d // 3)
    emb_h = learned_table(h, d // 3)
    emb_w = learned_table(w, d // 3)
    return concat(broadcast(emb_t, emb_h, emb_w), axis=-1)  # (t, h, w, d)


# Pixel-attention gate: multiplicative, no additive vector  [arxiv:2105.13677 §2.4] [eq. 8]
def pixel_attention(grid):
    # Per-channel gate from a 3x3 depth-wise conv with padding 1  [eq. 8]
    gate = sigmoid(depthwise_conv3x3(grid))       # (B, C, H, W)
    return grid * gate                            # (B, C, H, W)


# ALiBi: a distance penalty added to attention scores  [arxiv:2211.05100 §3.2]
# Output: (H, n, n), added to the scores before softmax
def alibi_bias(num_heads, n):
    # One fixed slope per head, from the figure caption  [fig. caption §3.2]
    slopes = [2 ** (-8 * i / num_heads) for i in range(1, num_heads + 1)]
    # Key position minus query position; zero or negative for causal masks
    dist = arange(n)[None, :] - arange(n)[:, None]   # (n, n)
    # Penalty grows with the distance  [arxiv:2211.05100 §3.2]
    return slopes[:, None, None] * minimum(dist, 0)  # (H, n, n)
# For non-power-of-two head counts the released code uses a different slope
# rule; this sketch follows the caption  [code: megatron/model/transformer.py#L596-L607]


# Rotary frequencies, one per pair of dimensions  [arxiv:2308.12950 §2.4]
# base is 10^4 in Llama 2 and 10^6 after long-context fine-tuning
def rope_frequencies(head_dim, base):
    return [base ** (-2 * i / head_dim) for i in range(head_dim // 2)]


# Random-walk positional encoding, precomputed once per graph  [eq. 10]
# Output: (n, k), return probability of each node after 1 .. k steps
def rwpe(adjacency, degrees, k):
    # Random-walk operator A D^-1; degrees clipped to at least 1
    # [arxiv:2110.07875 §3.1] [code: data/molecules.py#L182-L184]
    walk = adjacency @ diag(1 / max(degrees, 1))    # (n, n)
    power = walk
    columns = [diagonal(power)]
    for _ in range(k - 1):
        power = power @ walk
        columns.append(diagonal(power))
    return stack(columns, axis=1)                   # (n, k)
```

## Contradictions

- **Within [TransMIL](/papers/2106.00908):** Appendix B adds a positional
  embedding to the input. The code has no additive embedding, and the card reads
  PPEG as the only position encoding [arxiv:2106.00908 app. B]
  [code: models/TransMIL.py#L77-L80].
- **Within [TranAD](/papers/2201.07284):** the paper cites the standard sinusoidal
  form. The released code adds the sine and cosine of one angle to every
  dimension [arxiv:2201.07284 §3.3] [code: src/dlutils.py#L200-L206].
- **Within [CvT](/papers/2103.15808):** the text says removing the convolutional
  embedding "drops 0.8%" [arxiv:2103.15808 §4.4]; table 6 gives 81.6 against
  80.7, a 0.9 drop [arxiv:2103.15808 tab. 6]. The card also notes other text and
  table mismatches. Use the table values.
- **Within [Code Llama](/papers/2308.12950):** the paper's long-context base is
  10^6 [arxiv:2308.12950 §2.4]. The code's default is 10000, and the parameters
  that set 10^6 are not in the repository, so the released value is unconfirmed
  [code: llama/model.py#L35].
- **Within [Group-Free 3D](/papers/2104.00678):** the paper adds the refined
  encoding to each stage's output and passes it on [arxiv:2104.00678 §3.2]. The
  released code adds it inside each attention layer, to the query in
  self-attention and to the query and key in cross-attention
  [code: models/transformer.py#L59-L66]. The two placements give different
  networks, so say which one you implement.
- **Within [LSPE](/papers/2110.07875):** Eq. 12 treats the orthogonality term of
  the Laplacian loss as differentiable. The released code detaches it, so only
  the trace term gives gradients. The code also normalises the loss by batch
  and node counts, which the paper does not state [arxiv:2110.07875 §3.1]
  [eq. 12] [code: nets/ZINC_graph_regression/gatedgcn_net.py#L154-L158]. The
  card's ZINC gain from the loss (0.093 to 0.090) therefore cannot come from the
  orthogonality term in the code.
- **Within [OrientedFormer](/papers/2409.19648):** the paper uses plain content
  features as the value in self-attention [arxiv:2409.19648 eq. 10]. The released
  code passes content plus positional encoding as the single attention input
  [code: projects/OrientedFormer/orientedformer/orientedformer_decoder_layer.py#L184-L187].
  The card notes that the value tensor depends on an mmcv default that is not in
  the repository, so the card cannot confirm which one the code uses.
- **Within [Prithvi-EO-2.0](/papers/2412.02732):** the paper encodes latitude,
  longitude, year and day-of-year with 1D sine and cosine codes [arxiv:2412.02732 §3.2].
  The README says the metadata is encoded with 2D sine and cosine, with separate
  learned weights for the encoder and the decoder [code: README.md]. The card
  cannot say which version the released weights use.
- **Within [BLOOM](/papers/2211.05100):** the caption formula gives one set of
  slopes for 112 heads, and the reference code gives another
  [fig. caption §3.2] [code: megatron/model/transformer.py#L596-L607]. The card
  cannot say which one the released weights used.
- **Within [the tutorial](/papers/2205.01138):** the sinusoid base is 10,000 in
  the text and 1,000 in the figure [arxiv:2205.01138 §2.5.1] [fig. 8].
- **Within [TESSERA](/papers/2506.20380):** the main text calls the day-of-year
  encoding learnable, and the appendix describes fixed sine and cosine of the
  normalised day. The released inference code uses fixed sinusoids of the raw
  integer day [arxiv:2506.20380 §3.2] [app. Pre-training Details]
  [code: tessera_infer/src/models/modules.py#L58-L71]. The card cannot say which
  form produced the released embeddings.
- **Within [UCPE](/papers/2512.07237):** the text lists the latitude first and
  then the up vector. The released code concatenates them in the opposite order
  [arxiv:2512.07237 §3.2] [code: diffsynth/pipelines/wan_video_panshot.py#L953].
  Weights trained with one order do not load correctly with the other.
- **Within [Graph Transformers: A Survey](/papers/2407.09777):** the Laplacian
  eigenvector encoding is attributed to Rampášek et al. in §3.1.1, and a global
  spectral encoding is attributed to Graphormer in §3.1.2. The survey does not
  explain the split, so treat them as one idea with two sources
  [arxiv:2407.09777 §3.1.1] [arxiv:2407.09777 §3.1.2].
- **Within [iRPE](/papers/2107.14222):** the text gives α : β : γ = 1 : 2 : 8
  and says β sets the bucket count, but the DeiT configuration scales all three
  by r = 1.9 [arxiv:2107.14222 §4.1] [code: iRPE/DeiT-with-iRPE/rpe_models.py#L118].
  The Quantization mapping takes square roots in the paper [arxiv:2107.14222
  eq. 21], while the code applies the index to the squared distance
  [code: iRPE/DeiT-with-iRPE/irpe.py#L171]. Use the code's values to reproduce
  the reported numbers.
- **Within [StyleGAN-V](/papers/2112.14683):** the appendix sets the minimum
  period to 2³ frames and spaces the periods linearly, while the code defaults to
  16 frames and spaces them geometrically [arxiv:2112.14683 app. B]
  [arxiv:2112.14683 eq. 6] [code: configs/model/stylegan-v.yaml#L40]
  [code: src/training/motion.py#L218]. The code also has no amplitudes and takes
  sine and cosine at 256 frequencies [arxiv:2112.14683 eq. 2]
  [code: src/training/motion.py#L183].
- **Within [StyleGAN-V](/papers/2112.14683):** the paper encodes the discriminator's
  frame gap with a positional encoding, a two-layer MLP and a projection
  [arxiv:2112.14683 §3.2]. The code uses a learned embedding over the integer gap,
  concatenated with log-spaced Fourier features of the gap
  [code: src/training/layers.py#L255] [code: src/training/networks.py#L612].
- **Within [FLASH](/papers/2202.10447):** equation 5 and the listing in figure 2
  show no rotary embedding, while the appendix listings apply RoPE to queries
  and keys, and the hyperparameter tables list RoPE [arxiv:2202.10447 eq. 5]
  [arxiv:2202.10447 fig. 2] [arxiv:2202.10447 tab. 8] [arxiv:2202.10447 app. D].
  The listings also scale the scores by the sequence or chunk length, which the
  equations do not. Say which form you implement.
- **Within [SimTrack](/papers/2203.05328):** the paper defines R_ij as the
  fraction of the target area in each exemplar patch [arxiv:2203.05328 §3.2].
  The released code instead uses the IoU between each patch box and the target
  box, mapped to [0.5, 1] [code: lib/models/stark/clipvit16.py#L93-L96]. A
  literal reading of the paper gives different embeddings.
- **Within [BungeeNeRF](/papers/2112.05504):** the paper's encoding formula has
  only the damped sine and cosine terms [arxiv:2112.05504 §3.1]. The code keeps
  the raw mean as well [code: run_nerf_helpers.py#L59-L61]. The two give
  different input sizes.
- **Between these papers:** none known. The cards compare encodings only inside
  each paper's own ablations, so no contradiction can be drawn across them.

## Papers

- [Transformers in Time-series Analysis: A Tutorial](/papers/2205.01138): the
  fixed sinusoidal encoding, its offset property and its use in time-series
  variants; no experiments.
- [Robust Motion In-betweening](/papers/2102.04942): a sinusoidal code of frames
  remaining to a target, added to the encoder outputs, with a cumulative
  ablation.
- [Editable Free-viewpoint Video Using a Layered Neural Representation](/papers/2104.14786):
  a frequency encoding of position, direction and time in per-layer radiance
  field MLPs, with no ablation of the encoding.
- [VideoGPT: Video Generation using VQ-VAE and Transformers](/papers/2104.10157):
  learned spatio-temporal embeddings in a video VQ-VAE and a learned embedding in
  the code prior; no ablation of either.
- [Group-Free 3D Object Detection via Transformers](/papers/2104.00678): learned
  box and point embeddings per decoder layer, refreshed from each stage's
  prediction, with an ablation.
- [BARF: Bundle-Adjusting Neural Radiance Fields](/papers/2104.06405): a
  coarse-to-fine window on the NeRF frequency bands, opened during training, so
  that poses and scene are fitted together; compared with full encoding and with
  no encoding.
- [ResT: An Efficient Transformer for Visual Recognition](/papers/2105.13677):
  a depth-wise convolution gate in place of an additive encoding, with an ablation
  against a learned table and a group linear map.
- [CvT: Introducing Convolutions to Vision Transformers](/papers/2103.15808):
  no position embedding, with convolutional token embeddings and projections;
  an ablation of the position embedding on ImageNet-1k.
- [OrientedFormer](/papers/2409.19648):
  a Gaussian encoding of oriented boxes, with ablations against several other
  encodings.
- [Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications](/papers/2412.02732):
  a 3D sinusoidal encoding of the patch grid and sinusoidal metadata codes, with
  no ablation of the encoding.
- [TransMIL](/papers/2106.00908):
  a convolutional encoder over a square grid of patch tokens (PPEG), with an
  ablation against no encoding, sinusoids and plain convolutions.
- [TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data](/papers/2201.07284):
  a sinusoidal encoding of the window before the first encoder; the card sees no
  ablation of it.
- [LLaMA: Open and Efficient Foundation Language Models](/papers/2302.13971):
  rotary embeddings on queries and keys in every layer, with no absolute
  position embedding; no ablation of the encoding.
- [Code Llama: Open Foundation Models for Code](/papers/2308.12950): rotary
  embeddings with the base raised from 10^4 to 10^6 for long-context fine-tuning.
- [Transformer Quality in Linear Time](/papers/2202.10447): rotary embeddings
  over global positions across chunks, plus a relative bias in the chunk-local
  attention; the card has no ablation of the position terms.
- [Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis](/papers/2412.04431):
  RoPE2d on each scale, plus a learnable scale embedding; no ablation of the
  position encoding.
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](/papers/2211.05100):
  ALiBi distance penalties in place of position embeddings.
- [HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding](/papers/2601.14724):
  renumbering of cached positions after token pruning, with lazy and eager
  timings and an ablation against the base model.
- [CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows](/papers/2107.12571):
  a 2-D sine and cosine location code that conditions each scale's flow; the
  only ablation removes the whole conditioning.
- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](/papers/2207.08051):
  year, month and hour codes for image sequences, and a sinusoidal group code
  for band groups, with an ablation of the group code.
- [TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysis](/papers/2506.20380):
  a day-of-year encoding on each observation, with conflicting descriptions of
  its form and no ablation of it.
- [Unified Camera Positional Encoding for Controlled Video Generation](/papers/2512.07237):
  per-token camera-ray frames with a rotary encoding on half the channels, and
  an optional latitude-up map, with ablations against PRoPE and GTA.
- [Graph Neural Networks with Learnable Structural and Positional Representations](/papers/2110.07875):
  a learnable positional stream updated in every layer (LSPE), initialised from
  RWPE or LapPE, with ablations on ZINC and OGB molecular sets.
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](/papers/2104.00674):
  a frequency encoding of the surface point for its shape network (six bands)
  and its albedo network (ten bands); the reported results are for the full
  pipeline.
- [Graph Transformers: A Survey](/papers/2407.09777):
  a taxonomy of node positional and edge structural biases for graph
  transformers; a survey with no experiments of its own.
- [Rethinking and Improving Relative Position Encoding for Vision Transformer](/papers/2107.14222):
  a learned bucket table in the attention scores, indexed by row and column
  offset, with the Product mapping; ablations on DeiT-S and DETR.
- [StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2](/papers/2112.14683):
  an acyclic time embedding from a convolutional noise trajectory, with an
  ablation of the motion generator and of the discriminator's time condition.
- [BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering](/papers/2112.05504):
  the integrated positional encoding of Mip-NeRF, kept fixed, with training
  stages that add scales and residual blocks; the ablations remove the training
  scheme, not the encoding.
- [Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking](/papers/2203.05328):
  a learned embedding for the exemplar tokens from grid position and target
  overlap, with a foveal window; a cumulative ablation on TNL2K.
- [Introduction to Transformers: an NLP Perspective](/papers/2311.17633):
  a tutorial on the sinusoidal encoding added to the word embedding, with no
  experiments.