Positional encoding

Overview

Attention compares every position with every other position, so by itself it cannot tell which element came first. Positional encoding supplies that information, or a substitute for it such as the box geometry of a detected object. Each card here does it differently:

• Fixed sinusoids added to the input embedding, one vector per token index. This is the original Transformer scheme that the tutorials describe 2205.01138 §2.5.1 2311.17633 §2.2. Satellite and video models reuse the same sine and cosine form for each axis of a patch grid 2412.02732 §3.2 eq. 1.
• Learned embeddings per axis, shared by the attention layers of a video VQ-VAE and added once to the prior's inputs 2104.10157 §3.
• A sinusoidal code of time remaining to a target. The same formula is used, but the input is a countdown, not a token index 2102.04942 §3.3.
• Learned box and point embeddings inside each decoder layer. Each layer embeds the current box of every candidate and the coordinates of every point, and the boxes are refreshed from the previous stage 2104.00678 §3.2.
• A Gaussian sinusoidal encoding of an oriented box. The box is treated as a 2-D Gaussian, and its angle enters through the covariance 2409.19648 §3.3.
• A learned embedding for the exemplar tokens of a joint tracker. It is computed from each patch's grid position and its overlap with the target box. The search tokens keep the pre-trained embedding 2203.05328 §3.2 eq. 7.
• The integrated encoding of a conical frustum, with its bands fixed. Detail at each zoom level comes from training stages, not from a change to the encoding 2112.05504 §3.1 2112.05504 §3.3.
• A frequency encoding of position, direction and time in a radiance field. The encoding is applied to the inputs of each layer's MLP, and time enters it instead of a per-frame latent code 2104.14786 §4.2.
• A coarse-to-fine window on the same sine and cosine bands. A radiance field starts with only its smooth bands open, and opens the rest during training, so camera poses can be fitted together with the scene 2104.06405 §3.4.
• Frequency bands on the input of a signed-distance and an albedo network. Both networks encode the surface point with a frequency encoding, six bands for the shape and ten for the material 2104.00674 §3.
• A per-channel gate on the feature grid. Nothing is added to the input. Each value is multiplied by a sigmoid of a depth-wise convolution over its neighbours 2105.13677 §2.4 2105.13677 eq. 8.
• No position encoding at all. Overlapping convolutional token embeddings and depth-wise convolutional projections carry the local structure instead 2103.15808 §3.3 tab. 5.
• A distance penalty added to attention scores. Nothing is added to the embeddings. The penalty grows with the distance between query and key, with a fixed slope per head 2211.05100 §3.2.
• A convolutional encoder over the sequence, reshaped to a square grid. The grid is summed with three convolutions of itself 2106.00908 §3.3 alg. 3.
• Rotary embeddings on queries and keys, in every layer, with no table of position vectors 2302.13971 §2.2.
• Rotary embeddings over global positions inside a chunked linear-time attention, plus a relative bias. Each chunk is attended exactly, and the rotation is not restarted per chunk 2202.10447 §3.2 [arxiv:2202.10447 app. D].
• A 2-D sine and cosine location code as a condition, not added to the input. Each scale's normalising flow reads the code of its patch's row and column 2107.12571 §4.2.
• Position, date and band codes in a satellite model. Each token gets a position code, a year, month and hour code, and a group code for its band group 2207.08051 §4.1.1 2207.08051 §4.2.
• A day-of-year code on each observation in a per-pixel time series model 2506.20380 §3.2.
• A per-token camera-ray frame, with a rotary encoding on the other half of the channels, and an optional latitude-up map for pitch and roll 2512.07237 §3.2 2512.07237 §3.3.
• A learned bucket table added to the attention scores. Nothing is added to the patch embeddings. Each pair of patches is mapped to a bucket by its row and column offset, and each bucket holds a learned vector 2107.14222 §3.2 2107.14222 eq. 15. This is a relative encoding, and the wider comparison is on relative-positional-encoding.
• An acyclic sinusoidal time code from a learned trajectory. A one-dimensional convolution over sparse noise sets the wave parameters, so a video can be read at any real-valued time 2112.14683 §3.1 2112.14683 eq. 3.
• A learnable positional vector per node, updated in every layer. It starts from random-walk return probabilities and is kept apart from the structural features, so a graph network can tell apart nodes whose neighbourhoods match 2110.07875 §3.1.
• A taxonomy of node positional bias for graph transformers. Local options use hop distances, and global options use Laplacian eigenvectors or random-walk probabilities. It is a survey's classification, not a measured comparison 2407.09777 §3.1.1.

Encodings that enter through the attention score itself, relative to the query position, are covered on relative-positional-encoding, which also discusses rotary embeddings as used by PaLM. The iRPE bucket table and the StyleGAN-V time code have their own sections below. The rotary subsection below covers the LLaMA and Code Llama cards. The RoPE2d that one card here uses per scale 2412.04431 §3.5 is described on neither page.

Variants

Fixed sinusoidal encoding added to the input

Each position gets a vector of size d. For each pair of entries 2i and 2i+1, the vector holds the sine and cosine of the position divided by 10000^{2i/d} 2205.01138 §2.5.1 eq. 8 eq. 9. The encoding is summed with the token embedding before the first encoder block 2205.01138 §2.4.4. It has no learnable parameters 2205.01138 §2.5.1.

The tutorial argues that a fixed offset k maps the encoding at position i to the encoding at i+k by a rotation that depends only on k, so the encoding carries relative position 2205.01138 §2.5.3 eq. 13. That argument shows the shift is a linear map. It does not show that a trained model uses it 2205.01138 §2.5.3. The tutorial justifies summing rather than concatenating by analogy, not proof 2205.01138 §2.5.4.

The base is 10,000 in the text, which says it may change with sequence length without giving a rule, but 1,000 in the illustration 2205.01138 §2.5.1 fig. 8. The tutorial's time-series variants keep this encoding and change the input layer around it 2205.01138 §3. The tutorial runs no experiments, so it gives no benchmark for this variant.

A second tutorial, on the Transformer for NLP, gives the same sine and cosine formula with base 10000 and adds it to the word embedding [arxiv:2311.17633 §2.2]. It states that attention alone ignores word order, which is the reason for the encoding 2311.17633 §2.2. Like the first, it runs no experiments 2311.17633 §1. The two agree on the formula and the base, so they add no variant of their own.

The satellite foundation model Prithvi-EO-2.0 uses the same sine and cosine form in three dimensions. A 1D code is built for time, height and width, and the three are combined into one 3D encoding and added to the patch tokens 2412.02732 §3.2 2412.02732 eq. 1. Date and location metadata get their own 1D sine and cosine codes, each with a learned weight, and are added as a bias 2412.02732 §3.2. The paper gives the metadata drop probability as 0.1 for the TL models only 2412.02732 §3.2. The card reports no ablation of the positional part.

The anomaly detector TranAD adds the same sinusoidal form to the window before its first encoder 2201.07284 §3.3. Its ablations do not remove the encoding, so no benefit is measured 2201.07284 tab. 6. The released code adds both the sine and the cosine of one angle to every dimension, not one angle per sine-cosine pair, so the released encoding differs from the cited one dlutils.py.

Time-to-arrival code (sinusoid of frames remaining)

The motion in-betweening generator needs to know how far the target keyframe is. It encodes the number of frames until that keyframe with the sinusoidal formula, using dimension 256 and base 10,000 2102.04942 §3.2 2102.04942 §3.3. The code is held constant beyond a maximum that depends on the transition and past-context lengths 2102.04942 §3.3. The same code is added to all three encoder outputs before the LSTM 2102.04942 §1 2102.04942 §3.2.

On the Human3.6M walking benchmark, adding the code moves the cumulative average L2Q from 0.97 to 0.88 and average L2P from 2.52 to 2.04. The next step adds target noise and reaches 0.84 and 1.83 2102.04942 tab. 2. These are author-reported, and each row adds one component to the row above.

Learned axial embedding in a video VQ-VAE and prior (VideoGPT)

VideoGPT has no sinusoid. Its VQ-VAE uses learned spatio-temporal embeddings, shared by all axial attention layers in the encoder and decoder 2104.10157 §3. The released code learns one table per axis, each of size embedding divided by three, concatenates them and adds the result to the input attention.py. The paper says only "learned spatio-temporal embeddings" 2104.10157 §3. The prior's training step also adds a learned positional embedding to the shifted code embeddings gpt.py. The card reports no ablation of either embedding, so no benefit is measured.

Learned box and point embeddings per decoder layer (Group-Free 3D)

This model has no token order. Each candidate object attends to all seed points. Self-attention among candidates receives a learned embedding of the candidate's box, taken from the previous stage. Cross-attention to points receives a learned embedding of each point's coordinates. Every layer has its own embeddings 2104.00678 §3.2.

The paper says the refined encoding is added to each stage's output and fed to the next stage 2104.00678 §3.2. The released code adds it inside each attention layer instead. See Contradictions.

On ScanNet V2, with 6 layers and 256 candidates, averaged over 25 trials, the ablation reads as follows (mAP@0.25 / mAP@0.5) 2104.00678 tab. 5:

Reading the table: a fixed encoding barely changes the result, and refreshing the encoding from each stage's prediction accounts for most of the gain at IoU 0.5 2104.00678 tab. 5. The card says the paper gives no channel count, head count or query initialisation for this block 2104.00678 §3.2.

Joint exemplar-search tokens with a learned exemplar embedding (SimTrack)

SimTrack puts the exemplar tokens and the search tokens in one sequence. A separate learned embedding is added to the exemplar tokens only. It is computed from each patch's grid position (i, j) and a fraction R_ij of the target area in that patch, through two fully connected layers 2203.05328 §3.2 eq. 7. The search tokens keep the pre-trained embedding, because the search image has the same size as in standard ViT training 2203.05328 §3.2. The paper's reason for a separate embedding is that the exemplar is smaller than the search image, and one shared embedding would give the backbone no way to tell the two images apart 2203.05328 §3.2.

A foveal window adds 64×64 crops from the centre of the exemplar as extra tokens, cut into patches with their own positions 2203.05328 §3.3 app. Training Details. The paper does not describe any output for the foveal tokens, so they act as extra input only 2203.05328 §3.3.

The paper's cumulative ablation on TNL2K, for ViT-B/16 with STARK-SV as the base, reads as follows (AUC / precision) 2203.05328 tab. 8:

Each row adds one part to the row above, so the position embedding's +0.5 AUC is measured on top of the joint framework only. Table 4 compares the whole framework with STARK-SV on the same backbone. For ViT-B/16 the LaSOT AUC gain is +2.5 2203.05328 tab. 4. Results are author-reported, and the card has no separate test of the embedding alone.

Gaussian encoding of oriented boxes (OrientedFormer)

The detector predicts tilted rectangles, so the positional query is a box (x, y, w, h, θ). The box is turned into a 2-D Gaussian with mean (x, y) and a covariance that depends on the size and the angle 2409.19648 §3.3 2409.19648 eq. 2. The sinusoidal encoding is then evaluated as the expectation of a sine or cosine under that Gaussian. This equals the sine or cosine of the mean, damped by a factor that depends on the variance 2409.19648 eq. 4 2409.19648 eq. 5. Only the diagonal of the covariance is needed 2409.19648 eq. 6 2409.19648 eq. 7.

The same Gaussian geometry also biases decoder self-attention, through a Wasserstein distance between boxes 2409.19648 §3.4. That bias is a separate component, and the ablations below test it separately.

On DIOR-R, with ResNet-50 and AP50, the encoding ablation reads as follows 2409.19648 tab. 10:

The module ablation on DIOR-R combines the Gaussian encoding with the Wasserstein bias and with cross-attention 2409.19648 tab. 11. In that table, the Gaussian encoding alone with cross-attention reaches 67.03, against 65.78 for cross-attention alone. Both ablations come from one detector and one dataset.

Frequency encoding of position, direction and time (ST-NeRF)

ST-NeRF has no token order. Each layer is a function of a 3D position p, a view direction d and a time t. A deformation MLP predicts an offset from (p, t), and a radiance MLP predicts colour and density at the moved point 2104.14786 §4.2 eq. 2 eq. 3. The frame index enters through a positional encoding, not through a per-frame latent code 2104.14786 §4.2. The paper does not state the number of frequencies. The released code uses L = 10 for position and for the deformation input, and L = 4 for direction spacenet.py motion_net.py. The card reports no ablation of this encoding, so no benefit is measured.

Coarse-to-fine window on the frequency bands (BARF)

BARF uses the NeRF encoding of the previous variant: the raw coordinates, plus a sine and a cosine at each frequency 2^k \pi 2104.06405 eq. 10 2104.06405 eq. 11. Its window multiplies only the sine and cosine of band k. The window is zero while training progress \alpha is below k, rises along a half-cosine over one unit of progress, and is one once \alpha reaches k+1 2104.06405 eq. 13 2104.06405 eq. 14. The raw coordinates are never masked 2104.06405 §3.4. Progress grows linearly from iteration 20K to 100K of a 200K run, and all bands stay open after that [arxiv:2104.06405 §4.2] 2104.06405 §4.3.

The paper's reason for the window is that the derivative of a band scales with its frequency, so the high bands make the pose gradients hard to follow 2104.06405 §3.3 2104.06405 eq. 12. The window also closes the gradient of a masked band, so an open band has no effect on the poses until it opens 2104.06405 eq. 15.

The schedule is hand-set. The paper uses the same window for all LLFF scenes and calls a per-scene schedule out of scope 2104.06405 app. B.2. The released code treats the window as a fraction of the total iterations, so a shorter run opens the bands earlier barf.py.

On the eight Blender scenes with perturbed poses, mean rotation error falls from 6.167 to 0.193 degrees against full encoding, and PSNR rises from 22.12 to 27.50 dB 2104.06405 tab. 2. A NeRF with no positional encoding reaches 0.202 degrees and 26.78 dB on the same scenes. Against that baseline, the window adds little on rotation, and its PSNR gain is 0.72 dB 2104.06405 tab. 2. On the eight LLFF scenes from identity poses, BARF's rotation error is 0.573 degrees against 0.699 for the no-encoding baseline, but its translation error is 0.331 against 0.298, and its PSNR is 23.97 against 23.82 dB 2104.06405 tab. 3 2104.06405 tab. 5. The paper calls it competitive with that baseline rather than uniformly better. With metric depth in place of inverse depth, BARF's LLFF rotation error rises to 1.580 degrees, and the baseline is better on rotation, translation and PSNR 2104.06405 tab. 6.

Frequency bands on the surface point of an SDF and an albedo network (PhySG)

PhySG's geometry network is a signed distance function, and its albedo network gives the diffuse colour at a surface point. Both use a frequency encoding of that point. The shape network uses six frequencies and the albedo network ten 2104.00674 §3 default.conf default.conf. The paper gives the frequency form in a footnote 2104.00674 §3.

The paper's results come from the full pipeline, which also learns the lighting and the material. They do not isolate this encoding 2104.00674 tab. 1. The card gives no ablation of it.

Integrated encoding with scale-staged training (BungeeNeRF)

BungeeNeRF uses the integrated positional encoding of Mip-NeRF, and does not change it. Each sampled interval is a conical frustum, and its encoding is the sine and cosine of the frustum's mean at frequencies 2^m for m = 0 to 9, each damped by a factor that depends on the frustum's variance 2112.05504 §3.1 2112.05504 §4. The encoding is a fixed function of the frustum, and the scale of an image is handled by training. Training runs in stages, starting from the most distant scale. Each stage adds the next closer scale and one residual block with its own output, and each output is trained on all scales up to its own 2112.05504 §3.3 eq. 3.

The motivation is that close views need high-frequency bands and far views do not. Training one NeRF on all scales at once biases it toward low frequencies, so the paper argues close views blur 2112.05504 §3.2 fig. 2. Because the encoding stays fixed, the card cannot attribute any gain to it.

The ablations remove the training scheme, not the encoding. On 56 Leonard, the average PSNR is 23.481 for the full model, 22.139 without model growing and 22.340 without the data feeding scheme 2112.05504 tab. 3. Results are author-reported. The "same iter as baselines" row uses the baselines' budget, and the "until convergence" row trains longer, so the two rows are not like-for-like 2112.05504 tab. 2. The main baseline, Mip-NeRF, uses the same integrated encoding, so that comparison does not test the encoding either.

The released code keeps the raw mean as a first input feature, which the paper's formula omits. It gives 63 input dimensions for 10 bands run_nerf_helpers.py run_nerf_helpers.py. Use that size if you follow the code rather than the formula.

Depth-wise convolution gate, pixel attention (ResT)

ResT has no additive vector and no table tied to the input size. Each value is multiplied by a sigmoid of a 3x3 depth-wise convolution of the same grid 2105.13677 §2.4 2105.13677 eq. 8. The paper says this lets the network take any input size 2105.13677 §2.4. It compares this gate with a learned absolute table (LE) and a per-channel linear map (GL) 2105.13677 §2.4 2105.13677 app. A.4.

On ResT-Lite at 224 px, with the simple training recipe, the top-1 results read as follows 2105.13677 tab. 8:

The paper argues that PA encodes position and allows any input size 2105.13677 §2.4. The ablation compares only ResT-Lite at 224 px, so it does not test different input sizes 2105.13677 tab. 8. The reference code computes height and width from the input, with no learned table tied to the input size rest.py. The code's GL is not the paper's GL: the code's version is a residual depth-wise convolution, and no model in the repository calls it rest.py.

No position embedding, convolutions instead (CvT)

CvT drops the position embedding from all three stages. Its tokens come from an overlapping convolution, and each attention block computes queries, keys and values with depth-wise convolutions 2103.15808 §3.1 2103.15808 §3.2. The authors say local relations are modelled by these convolutions, which also helps with inputs of varying resolution 2103.15808 §3.3 2103.15808 §4.4.

On ImageNet-1k top-1, the position-embedding ablation reads as follows 2103.15808 tab. 5:

Table 6 removes the convolutional token embedding and crosses it with the position embedding. Without the convolutional embedding, top-1 is 80.7 with no position embedding and 81.1 with one. With it, the values are 81.4 with a position embedding and 81.6 without 2103.15808 tab. 6. So within CvT-13 the position embedding moves top-1 by 0.2 points or less. The paper's variable resolution claim is not tested on ImageNet 2103.15808 §4.4. The ablation is for the classification setting only.

Convolutional encoder over a square grid (PPEG, TransMIL)

Attention over a bag of patches has no grid, so TransMIL pads the sequence to a square, reshapes the patch tokens into a grid in input order, and sums the grid with three convolutions of it, with kernels 7, 5 and 3 2106.00908 alg. 3 2106.00908 §3.3. The convolutions are grouped per channel TransMIL.py. The model has two attention layers, and this step sits between them 2106.00908 alg. 2. "Spatial" here means sequence order, not (x, y) coordinates, so the spatial signal depends on how the feature files were written README.md.

On CAMELYON16 and NSCLC, scored by AUC, the position-encoding ablation reads as follows 2106.00908 tab. 2:

Every row with a position encoding beats "none", and PPEG is highest on both datasets 2106.00908 tab. 2. The paper reports no standard deviations §4. The "both" row is not defined in the text 2106.00908 tab. 2. The card cannot check the ablation, because its scripts are not in the repository README.md.

ALiBi attention bias (BLOOM)

BLOOM adds no position vector to its embeddings. For each head, it adds a term to the attention score that falls with the distance between key and query. The slope is fixed for each head 2211.05100 §3.2. The figure caption gives the slopes as 2^{-8i/n} for head i among n heads fig. caption §3.2.

The authors chose this over learned or rotary embeddings, based on ablations run at 1.3B parameters 2211.05100 §3.2. The card gives no numbers for that comparison. BLOOM has 112 heads, which is not a power of two. The reference code uses a different slope rule in that case, so the caption formula is not what the released model computed transformer.py. The paper does not say which rule the trained model used. The model was trained on sequences of 2048 tokens 2211.05100 §3.5, and the card cites no test beyond that length.

Rotary embeddings on queries and keys (LLaMA, Code Llama)

LLaMA rotates the query and key vectors of every attention layer by a position-dependent angle, and uses no absolute position embedding 2302.13971 §2.2. The paper does not state the rotation base or the context length. The released code uses a base of 10000 and rotates adjacent channel pairs, not the two halves of each head model.py model.py. Other codebases split the halves, which changes the attention scores relative to the code. The LLaMA card reports no ablation of the encoding.

Code Llama keeps the rotary form and raises the base from 10^4 to 10^6 for long-context fine-tuning on 16,384-token sequences, with a learning rate of 2e-5 2308.12950 §2.4. Frequencies follow base^(-2i/d), one per pair of dimensions model.py. The paper says the change is different from position interpolation, because the period of the rotation is changed instead of scaling the frequencies linearly 2308.12950 §2.4. The paper reports that the change can be applied to a pretrained model, and that the loss settles within a few steps at a low learning rate [arxiv:2308.12950 app. G].

Fine-tuning with the raised base improves single-line completion on LCC-balanced at 7B, from 36.86 to 39.23 exact match. It lowers HumanEval pass@1 by 0.52 points and MBPP pass@1 by 1.9 points on average 2308.12950 §3.3. These results are for code tasks only. RoPE itself has no card here; see relative-positional-encoding for PaLM's use of it.

Rotary positions across chunks, with a relative bias (FLASH)

FLASH cuts the sequence into chunks. Inside each chunk it computes exact squared-ReLU attention, and between chunks it uses linear attention 2202.10447 §3.2. The attention score carries a relative position bias b, which the paper calls a relative position bias but does not define 2202.10447 §2 eq. 5. The appendix listing applies rotary embeddings to queries and keys over the whole sequence, with the rotation not restarted in each chunk, so every token keeps its global position 2202.10447 app. D. The same listing builds b in one of two ways, chosen by the length argument: a learned table of 2n−1 weights below length 512, or a Toeplitz matrix from rotated vectors at 512 and above 2202.10447 app. D. The paper's hyperparameter tables list RoPE on queries and keys 2202.10447 tab. 8.

The position scheme is not measured on its own. The main comparison pits FLASH against Transformer++, which also uses rotary embeddings and a gated linear unit, while FLASH also changes the attention, the gate and the chunking 2202.10447 §4. At 8192 tokens on Wiki-40B, FLASH reaches perplexity 15.109 against 15.254 for Transformer++, at 62.5 against 261.0 ms per training step on 64 TPU-v4 cores 2202.10447 tab. 13. Those are author-reported results.

Sinusoidal location code as a flow condition (CFLOW-AD)

CFLOW-AD adds nothing to its features. Each scale has its own conditional normalising flow, and that flow is conditioned on a fixed 2-D code of each location, with 128 channels in all experiments 2107.12571 §4.2. The code is built from sine and cosine harmonics, and half of the channels encode the row and half the column model.py. The condition is concatenated inside each coupling layer 2107.12571 §4.2.

The paper gives no ablation of the code by itself. Its one flow comparison replaces the conditional flow with an unconditional one, which removes the condition as a whole. On MVTec AD with WRN50, 8 blocks and 3 scales at 512 px, the average localisation AUROC is 97.36 for the conditional flow and 96.86 for the unconditional one 2107.12571 tab. 2. The paper reports the conditional flow as about 0.5 points better on average 2107.12571 §5.2.

Year, month and hour codes with band-group codes (SatMAE)

SatMAE's temporal path gives each token two codes that are concatenated to the embedding width. The first is a positional code for the token's place in the image. The second is built from the year, month and hour of the token's image, and drops the day, minute and second 2207.08051 §4.1.1 2207.08051 eq. 2. In the temporal runs the positional code has 640 dimensions and the temporal code 384, split across the three fields 2207.08051 app. A.3.2. The paper does not give an origin for the year, so an implementation must choose one and keep it the same in pre-training and fine-tuning.

For multi-band input, the bands form groups, each with its own patch embedding. A sinusoidal code of the group index is concatenated to the positional code 2207.08051 eq. 3 2207.08051 §4.2. For the Sentinel-2 runs the positional code has 768 dimensions and the group code 256 2207.08051 app. A.3.3.

On fMoW-Sentinel validation, top-1 is 59.30 with the group code and 58.87 without it, with the same grouping and independent masking 2207.08051 §5.4. This is the cleanest isolated test of a code in the card. The temporal ablation is not as clean: the row without the temporal encoding (78.07) uses independent masking, and the row with it (78.45) uses consistent masking, so the encoding's own effect is not measured 2207.08051 §5.3.

Day-of-year encoding of observation dates (TESSERA)

TESSERA's encoder works on a set of valid observations for one pixel. Each sensor's encoder adds a day-of-year (DOY) encoding to each date's embedding 2506.20380 §3.2. The main text describes this encoding as learnable. The appendix describes sine and cosine of the normalised DOY [arxiv:2506.20380 §3.2] app. Pre-training Details. The ablation removes global shuffling, mixup, Sentinel-1, quantisation and pre-training, but not the DOY encoding 2506.20380 tab. 3. So the card has no measured effect for it.

Ray-frame encoding with a latitude-up map (UCPE)

UCPE gives each token one viewing ray, with the ray's origin and direction 2512.07237 eq. 5. A local frame is built for each ray, with the ray direction as the z-axis, the cross product of the camera's downward axis and z as the x-axis, and y completing the frame 2512.07237 eq. 6. Attention uses the inverse transform 2512.07237 eq. 7. The ray transform takes half of the feature dimensions and RoPE takes the other half, as one block-diagonal operator per token 2512.07237 eq. 10 2512.07237 §3.3. Lens effects are carried by the rays, not by a separate intrinsic matrix 2512.07237 §3.2.

The latitude-up map is optional. It gives each pixel a latitude and a two-part up vector, which a linear layer projects to the token width and adds as a bias 2512.07237 §3.2 2512.07237 §3.3. The paper uses a rotation of 0.1 rad to build the up vector 2512.07237 app. B.

The ray encoding is compared with two replacements inside the same adapter. With PRoPE, rotation error is 5.35 degrees and camera-motion error 20.58; with GTA, 5.27 and 20.14. The UCPE default gives 4.12 and 17.59 on the same UCM-trained setting 2512.07237 tab. 1. The latitude-up map is reported with its pitch and roll numbers only. Without it, rotation error is 4.29, but translation error (13.46 against 15.21) and camera-motion error (15.94 against 17.59) are lower, so the map does not improve those two metrics 2512.07237 tab. 1. The training data is synthetic, and the zero-shot RealEstate10K test uses 100 clips 2512.07237 app. A 2512.07237 tab. 2.

Learnable positional stream updated in every layer (LSPE, graph networks)

Message-passing graph networks update a node from its neighbours, so two nodes with the same neighbourhood stay alike 2110.07875 §1. The LSPE card adds a second per-node stream of position values. The structural update reads the concatenation of structural and positional features. The positional update is a separate message-passing function, and a tanh lets the coordinates take either sign 2110.07875 §3.1 eq. 7 eq. 8 eq. 9. The comparison point is a baseline that concatenates a fixed encoding to the input and never updates it 2110.07875 §3.1 eq. 3.

The initial vector is either random-walk return probabilities (RWPE) or Laplacian eigenvectors (LapPE). RWPE has no sign ambiguity. LapPE needs random sign flips during training 2110.07875 §3.1. The RWPE is precomputed before training 2110.07875 app. A.

On ZINC, GatedGCN with RWPE and the per-layer stream reaches test MAE 0.093 ± 0.003. Without any encoding it reaches 0.251 ± 0.009, and with LapPE 0.202 ± 0.006. The Laplacian auxiliary loss lowers the first value to 0.090 ± 0.001 2110.07875 tab. 1. Injecting RWPE at the input without the per-layer stream gives 0.122 ± 0.003 2110.07875 tab. 3. Most of the gain in the card therefore needs the per-layer update, not the RWPE alone. The numbers are author-reported, averaged over 4 seeds 2110.07875 §4.1.

On the two OGB molecular sets the changes are small. GatedGCN with RWPE reaches 0.775 ± 0.003 test ROC-AUC on MOLTOX21, against 0.772 ± 0.006 without an encoding. The two runs' spreads overlap 2110.07875 tab. 1. The card reads these as comparable, not as wins 2110.07875 §4.2.

Node positional bias in graph transformers (survey taxonomy)

A graph transformer can add a vector to each node that says where the node sits in the graph. The survey sorts these into local and global options 2407.09777 §3.1.1. Local options are one-hot hop distances, learned embeddings of edge types, and graph-kernel similarities between neighbourhoods. Global options are Laplacian eigenvectors or eigenvalues, random-walk or diffusion probabilities such as personalised PageRank and the heat kernel, and a low-dimensional embedding that keeps graph distance 2407.09777 §3.1.1.

Edge structure enters attention in a related way, as a bias on the scores or as extra tokens. The survey's examples include a shortest-path kernel and global spectral encodings 2407.09777 §3.1.2. It also places a message-passing step before, between or after the attention layers 2407.09777 §3.1.3.

The survey is a classification, and its tables give no performance numbers 2407.09777 §4. Its only numeric guidance is a design guide, which lists adding structural and positional encodings as one principle 2407.09777 §6. Use the original papers for any measured effect.

Renumbering positions after cache compaction (HERMES)

This is not an encoding. A streaming video model that drops cached tokens keeps the positions of the survivors in range. HERMES renumbers them when positions approach the model's maximum, and corrects the cached keys with a rotary delta instead of recomputing them 2601.14724 §3.3 app. E. Lazy renumbering runs only near the limit, and eager renumbering runs at every compression step 2601.14724 §4.1 2601.14724 §4.4. The paper does not state the margin before the limit. The reference code starts compaction 1024 positions early llavaov_hermes.py.

The ablation compares lazy and eager renumbering with the base model. On StreamingBench real-time and OVO-Bench, the OVO average is 56.94 for lazy and 56.06 for eager, against 53.35 for the base model 2601.14724 tab. 8. On VideoMME, the average is 54.93 for lazy and 58.44 for eager, against 57.67 for the base model 2601.14724 tab. 9. The table has no row that compacts the cache without renumbering, so the renumbering effect is not isolated.

Learned bucket table in the attention scores (iRPE)

iRPE adds a learned term b_{ij} to each attention logit before the division by \sqrt{d} 2107.14222 eq. 13. The bucket of a pair of patches comes from a piecewise index on its row offset and on its column offset. The index is exact up to \alpha, log-spaced up to \gamma, and capped at \beta 2107.14222 eq. 18. The Product mapping gives one bucket per (row bucket, column bucket) pair, and the paper's main setting is contextual mode, where each bucket vector is dotted with the query and the table is shared across heads 2107.14222 §3.2 tab. 2. The paper sets the table to zero at the start 2107.14222 supp. Weight Initialization.

On DeiT-S, ImageNet top-1, the rows that matter read as follows 2107.14222 tab. 4:

The paper reads these as relative encodings replacing the absolute one for classification. Detection is different. On COCO with DETR at 150 epochs, the sinusoidal encoding alone gives 39.5 AP, with no encoding 30.4, and the contextual 9×9 relative table alone 38.7. With both it gives 40.8 2107.14222 tab. 6 rows 1, 2, 4, 6. The authors also report at most 1% extra MACs for DeiT-S across the resolutions they test 2107.14222 §4.2 fig. 4. All numbers are author-reported.

Acyclic time embedding for a continuous-time video generator (StyleGAN-V)

StyleGAN-V has no token index. Its input is a time, and a clip can be read at any real-valued time. The authors argue that a standard sine embedding is periodic, so a generated video repeats itself 2112.14683 §3.1. The motion noise sits on a grid of times spaced \delta^z apart, with 16 frames by default and 256 for SkyTimelapse. Two padding-free one-dimensional convolutions with kernel 11 and hidden size 512 turn that noise into a trajectory 2112.14683 app. B. For a query time, the grid code on its left sets the amplitudes, phases and periods of the waves. A stitching term makes the embedding start and end at the grid points, and alignment vectors restore the expressiveness that stitching removes 2112.14683 §3.1 2112.14683 eq. 4. The resulting 512-dimensional code is tiled into the constant input of the StyleGAN2 synthesis network 2112.14683 fig. 5.

The paper does not test the sinusoidal form by itself. Its motion ablation swaps the convolutional trajectory for LSTM codes. On FaceForensics, FVD16 is 47.41 by default, 180.3 with LSTM codes at \delta^z = 16, and 131.9 at \delta^z = 1 2112.14683 tab. 2. That changes the whole motion generator, so it does not isolate the embedding. The authors list remaining periodic motion as a limit 2112.14683 app. A.

The discriminator also receives the gap between the frames of a clip as a time condition. Removing it raises FaceForensics FVD16 from 47.41 to 95.4, and SkyTimelapse FVD16 from 79.52 to 102.1 2112.14683 tab. 2. The form of that encoding differs between paper and code (see Contradictions).

When to use which

Most cards give ablations for only some of these variants, and each ablation is measured inside its own system. The strongest comparisons are the time-to-arrival code, the iterative box encoding, the Gaussian oriented-box encoding, the ResT gate and the PPEG table. Choose by the constraint, then check the card before trusting a benefit.

Implementation sketch

The shared core is the fixed sinusoid. The other variants change what goes in (a countdown, a box, a learned table), where it goes (per layer, or into the score), or both.

# Fixed sinusoidal encoding, added to the input embedding  [arxiv:2205.01138 §2.5.1]
# Output: (n, d) with one row per position
def sinusoidal_encoding(n, d, base=10000):
    enc = zeros(n, d)
    for pos in range(n):
        for i in range(d // 2):
            # One frequency per sine/cosine pair  [eq. 8] [eq. 9]
            angle = pos / base ** (2 * i / d)
            enc[pos, 2 * i] = sin(angle)
            enc[pos, 2 * i + 1] = cos(angle)
    return enc
# x = token_embedding + sinusoidal_encoding(n, d)  [arxiv:2205.01138 §2.4.4]


# Time-to-arrival code: the same function applied to a countdown  [arxiv:2102.04942 §3.3]
# pos = min(frames_left, cap); held constant past the cap, cap value derived in card
# code = sinusoidal_encoding(1, 256)[0] with pos replaced by that value


# Learned per-axis embedding, concatenated across axes  [code: videogpt/attention.py#L458-L493]
# One table per axis, each of size d / 3, then concatenated and added to the input
def axial_embedding(t, h, w, d):
    emb_t = learned_table(t, d // 3)
    emb_h = learned_table(h, d // 3)
    emb_w = learned_table(w, d // 3)
    return concat(broadcast(emb_t, emb_h, emb_w), axis=-1)  # (t, h, w, d)


# Pixel-attention gate: multiplicative, no additive vector  [arxiv:2105.13677 §2.4] [eq. 8]
def pixel_attention(grid):
    # Per-channel gate from a 3x3 depth-wise conv with padding 1  [eq. 8]
    gate = sigmoid(depthwise_conv3x3(grid))       # (B, C, H, W)
    return grid * gate                            # (B, C, H, W)


# ALiBi: a distance penalty added to attention scores  [arxiv:2211.05100 §3.2]
# Output: (H, n, n), added to the scores before softmax
def alibi_bias(num_heads, n):
    # One fixed slope per head, from the figure caption  [fig. caption §3.2]
    slopes = [2 ** (-8 * i / num_heads) for i in range(1, num_heads + 1)]
    # Key position minus query position; zero or negative for causal masks
    dist = arange(n)[None, :] - arange(n)[:, None]   # (n, n)
    # Penalty grows with the distance  [arxiv:2211.05100 §3.2]
    return slopes[:, None, None] * minimum(dist, 0)  # (H, n, n)
# For non-power-of-two head counts the released code uses a different slope
# rule; this sketch follows the caption  [code: megatron/model/transformer.py#L596-L607]


# Rotary frequencies, one per pair of dimensions  [arxiv:2308.12950 §2.4]
# base is 10^4 in Llama 2 and 10^6 after long-context fine-tuning
def rope_frequencies(head_dim, base):
    return [base ** (-2 * i / head_dim) for i in range(head_dim // 2)]


# Random-walk positional encoding, precomputed once per graph  [eq. 10]
# Output: (n, k), return probability of each node after 1 .. k steps
def rwpe(adjacency, degrees, k):
    # Random-walk operator A D^-1; degrees clipped to at least 1
    # [arxiv:2110.07875 §3.1] [code: data/molecules.py#L182-L184]
    walk = adjacency @ diag(1 / max(degrees, 1))    # (n, n)
    power = walk
    columns = [diagonal(power)]
    for _ in range(k - 1):
        power = power @ walk
        columns.append(diagonal(power))
    return stack(columns, axis=1)                   # (n, k)

Contradictions

• Within TransMIL: Appendix B adds a positional embedding to the input. The code has no additive embedding, and the card reads PPEG as the only position encoding 2106.00908 app. B TransMIL.py.
• Within TranAD: the paper cites the standard sinusoidal form. The released code adds the sine and cosine of one angle to every dimension 2201.07284 §3.3 dlutils.py.
• Within CvT: the text says removing the convolutional embedding "drops 0.8%" 2103.15808 §4.4; table 6 gives 81.6 against 80.7, a 0.9 drop 2103.15808 tab. 6. The card also notes other text and table mismatches. Use the table values.
• Within Code Llama: the paper's long-context base is 10^6 2308.12950 §2.4. The code's default is 10000, and the parameters that set 10^6 are not in the repository, so the released value is unconfirmed model.py.
• Within Group-Free 3D: the paper adds the refined encoding to each stage's output and passes it on 2104.00678 §3.2. The released code adds it inside each attention layer, to the query in self-attention and to the query and key in cross-attention transformer.py. The two placements give different networks, so say which one you implement.
• Within LSPE: Eq. 12 treats the orthogonality term of the Laplacian loss as differentiable. The released code detaches it, so only the trace term gives gradients. The code also normalises the loss by batch and node counts, which the paper does not state 2110.07875 §3.1 eq. 12 gatedgcn_net.py. The card's ZINC gain from the loss (0.093 to 0.090) therefore cannot come from the orthogonality term in the code.
• Within OrientedFormer: the paper uses plain content features as the value in self-attention 2409.19648 eq. 10. The released code passes content plus positional encoding as the single attention input orientedformer_decoder_layer.py. The card notes that the value tensor depends on an mmcv default that is not in the repository, so the card cannot confirm which one the code uses.
• Within Prithvi-EO-2.0: the paper encodes latitude, longitude, year and day-of-year with 1D sine and cosine codes 2412.02732 §3.2. The README says the metadata is encoded with 2D sine and cosine, with separate learned weights for the encoder and the decoder README.md. The card cannot say which version the released weights use.
• Within BLOOM: the caption formula gives one set of slopes for 112 heads, and the reference code gives another fig. caption §3.2 transformer.py. The card cannot say which one the released weights used.
• Within the tutorial: the sinusoid base is 10,000 in the text and 1,000 in the figure 2205.01138 §2.5.1 fig. 8.
• Within TESSERA: the main text calls the day-of-year encoding learnable, and the appendix describes fixed sine and cosine of the normalised day. The released inference code uses fixed sinusoids of the raw integer day 2506.20380 §3.2 app. Pre-training Details modules.py. The card cannot say which form produced the released embeddings.
• Within UCPE: the text lists the latitude first and then the up vector. The released code concatenates them in the opposite order 2512.07237 §3.2 wan_video_panshot.py. Weights trained with one order do not load correctly with the other.
• Within Graph Transformers: A Survey: the Laplacian eigenvector encoding is attributed to Rampášek et al. in §3.1.1, and a global spectral encoding is attributed to Graphormer in §3.1.2. The survey does not explain the split, so treat them as one idea with two sources 2407.09777 §3.1.1 2407.09777 §3.1.2.
• Within iRPE: the text gives α : β : γ = 1 : 2 : 8 and says β sets the bucket count, but the DeiT configuration scales all three by r = 1.9 2107.14222 §4.1 rpe_models.py. The Quantization mapping takes square roots in the paper [arxiv:2107.14222 eq. 21], while the code applies the index to the squared distance irpe.py. Use the code's values to reproduce the reported numbers.
• Within StyleGAN-V: the appendix sets the minimum period to 2³ frames and spaces the periods linearly, while the code defaults to 16 frames and spaces them geometrically 2112.14683 app. B 2112.14683 eq. 6 stylegan-v.yaml motion.py. The code also has no amplitudes and takes sine and cosine at 256 frequencies 2112.14683 eq. 2 motion.py.
• Within StyleGAN-V: the paper encodes the discriminator's frame gap with a positional encoding, a two-layer MLP and a projection 2112.14683 §3.2. The code uses a learned embedding over the integer gap, concatenated with log-spaced Fourier features of the gap layers.py networks.py.
• Within FLASH: equation 5 and the listing in figure 2 show no rotary embedding, while the appendix listings apply RoPE to queries and keys, and the hyperparameter tables list RoPE 2202.10447 eq. 5 2202.10447 fig. 2 2202.10447 tab. 8 2202.10447 app. D. The listings also scale the scores by the sequence or chunk length, which the equations do not. Say which form you implement.
• Within SimTrack: the paper defines R_ij as the fraction of the target area in each exemplar patch 2203.05328 §3.2. The released code instead uses the IoU between each patch box and the target box, mapped to [0.5, 1] clipvit16.py. A literal reading of the paper gives different embeddings.
• Within BungeeNeRF: the paper's encoding formula has only the damped sine and cosine terms 2112.05504 §3.1. The code keeps the raw mean as well run_nerf_helpers.py. The two give different input sizes.
• Between these papers: none known. The cards compare encodings only inside each paper's own ablations, so no contradiction can be drawn across them.

Papers

• Transformers in Time-series Analysis: A Tutorial: the fixed sinusoidal encoding, its offset property and its use in time-series variants; no experiments.
• Robust Motion In-betweening: a sinusoidal code of frames remaining to a target, added to the encoder outputs, with a cumulative ablation.
• Editable Free-viewpoint Video Using a Layered Neural Representation: a frequency encoding of position, direction and time in per-layer radiance field MLPs, with no ablation of the encoding.
• VideoGPT: Video Generation using VQ-VAE and Transformers: learned spatio-temporal embeddings in a video VQ-VAE and a learned embedding in the code prior; no ablation of either.
• Group-Free 3D Object Detection via Transformers: learned box and point embeddings per decoder layer, refreshed from each stage's prediction, with an ablation.
• BARF: Bundle-Adjusting Neural Radiance Fields: a coarse-to-fine window on the NeRF frequency bands, opened during training, so that poses and scene are fitted together; compared with full encoding and with no encoding.
• ResT: An Efficient Transformer for Visual Recognition: a depth-wise convolution gate in place of an additive encoding, with an ablation against a learned table and a group linear map.
• CvT: Introducing Convolutions to Vision Transformers: no position embedding, with convolutional token embeddings and projections; an ablation of the position embedding on ImageNet-1k.
• OrientedFormer: a Gaussian encoding of oriented boxes, with ablations against several other encodings.
• Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications: a 3D sinusoidal encoding of the patch grid and sinusoidal metadata codes, with no ablation of the encoding.
• TransMIL: a convolutional encoder over a square grid of patch tokens (PPEG), with an ablation against no encoding, sinusoids and plain convolutions.
• TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data: a sinusoidal encoding of the window before the first encoder; the card sees no ablation of it.
• LLaMA: Open and Efficient Foundation Language Models: rotary embeddings on queries and keys in every layer, with no absolute position embedding; no ablation of the encoding.
• Code Llama: Open Foundation Models for Code: rotary embeddings with the base raised from 10^4 to 10^6 for long-context fine-tuning.
• Transformer Quality in Linear Time: rotary embeddings over global positions across chunks, plus a relative bias in the chunk-local attention; the card has no ablation of the position terms.
• Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis: RoPE2d on each scale, plus a learnable scale embedding; no ablation of the position encoding.
• BLOOM: A 176B-Parameter Open-Access Multilingual Language Model: ALiBi distance penalties in place of position embeddings.
• HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding: renumbering of cached positions after token pruning, with lazy and eager timings and an ablation against the base model.
• CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows: a 2-D sine and cosine location code that conditions each scale's flow; the only ablation removes the whole conditioning.
• SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery: year, month and hour codes for image sequences, and a sinusoidal group code for band groups, with an ablation of the group code.
• TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysis: a day-of-year encoding on each observation, with conflicting descriptions of its form and no ablation of it.
• Unified Camera Positional Encoding for Controlled Video Generation: per-token camera-ray frames with a rotary encoding on half the channels, and an optional latitude-up map, with ablations against PRoPE and GTA.
• Graph Neural Networks with Learnable Structural and Positional Representations: a learnable positional stream updated in every layer (LSPE), initialised from RWPE or LapPE, with ablations on ZINC and OGB molecular sets.
• PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting: a frequency encoding of the surface point for its shape network (six bands) and its albedo network (ten bands); the reported results are for the full pipeline.
• Graph Transformers: A Survey: a taxonomy of node positional and edge structural biases for graph transformers; a survey with no experiments of its own.
• Rethinking and Improving Relative Position Encoding for Vision Transformer: a learned bucket table in the attention scores, indexed by row and column offset, with the Product mapping; ablations on DeiT-S and DETR.
• StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2: an acyclic time embedding from a convolutional noise trajectory, with an ablation of the motion generator and of the discriminator's time condition.
• BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering: the integrated positional encoding of Mip-NeRF, kept fixed, with training stages that add scales and residual blocks; the ablations remove the training scheme, not the encoding.
• Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking: a learned embedding for the exemplar tokens from grid position and target overlap, with a foveal window; a cumulative ablation on TNL2K.
• Introduction to Transformers: an NLP Perspective: a tutorial on the sinusoidal encoding added to the word embedding, with no experiments.