Fourier features

Overview

A network that reads its input directly has to build any fine or periodic pattern out of smooth functions of that input. Fourier features give it the sines and cosines of the input at chosen frequencies as extra channels 2107.00630 §3.4 2307.00379 §2.3.3. The same encoding turns a low-dimensional condition, such as an object's box, into a richer vector 2412.17226 eq. 2. It also encodes the coordinates that a neural field takes as input, such as a mesh vertex or a point and view direction in a radiance field 2112.03221 eq. 1 2011.10379 §3.1. None of the cards describes learned frequencies: they are set by hand, drawn from a Gaussian, or not stated. What changes between variants is the input, the frequencies and the reason for choosing them.

Variants

Multi-octave features on the denoiser input

The denoiser's input is the noisy latent with extra channels sin(2^n pi z) and cos(2^n pi z) for a range of integers n. The paper uses n in {7, 8} 2107.00630 app. C. Card: Variational Diffusion Models.

The authors report a much better likelihood with the features in their ablation 2107.00630 fig. 5. The same features gave no benefit when added to PixelCNN++, and the paper did not observe gains for autoregressive models in general 2107.00630 §6.2 2107.00630 §3.4. The reported numbers use learned schedule endpoints together with the features; with a fixed schedule, the model stays above 4 bits per dimension on the test set 2107.00630 §6.2.

Period-matched embedding for periodic boundaries

For a PINN with exactly periodic boundary conditions, the input coordinates are mapped through a Fourier embedding whose period P equals the domain length. The paper gives the one- and two-dimensional forms [arxiv:2307.00379 §2.3.3]; the basis itself is in its eq. 22 and is not reproduced here. Card: Residual-based attention in physics-informed neural networks.

In the paper's Allen-Cahn ablation, the same residual-based attention weighting gives relative L2 error 5.2e-3 alone and 1.0e-4 with the Fourier embedding 2307.00379 tab. 2. These are single-seed numbers, which the paper takes from its best seed 2307.00379 §3.1.1.

Truncated 2D tensor embedding for Helmholtz

For the 2D Helmholtz problem the paper uses a truncated embedding with m = n = 5 2307.00379 §3.2 2307.00379 eq. 31. The paper does not give the period for this case; the released code fixes it at 2 (see the card's Paper vs code section). The paper also says the embedding is partly a comparison, since it contains the solution's trigonometric structure 2307.00379 §3.2.1.

The five-seed means are 8.21e-6 for the Fourier variant and 7.73e-5 for the approximate-distance-function variant 2307.00379 tab. 3. The two differ in boundary handling as well: the Fourier variant keeps a boundary loss, and the other drops it 2307.00379 §3.2. The gap therefore does not isolate the embedding.

Random Fourier mapping of vertex positions for a neural style field

Text2Mesh maps each vertex position p to [cos(2 pi B p), sin(2 pi B p)], where the entries of B are drawn from a Gaussian with standard deviation sigma. The raw position is kept alongside these features 2112.03221 eq. 1 2112.03221 app. D.1. The paper gives the shape of B as n × 3 but does not define n, and it gives the encoding a 256-dimensional feature 2112.03221 app. D.1. The features feed a style field that outputs a colour and a bounded offset along the normal for each vertex 2112.03221 §3.1. Card: Text2Mesh.

The paper calls the frequency scale a hyperparameter that the user controls 2112.03221 §4.1 2112.03221 fig. 7, and suggests sigma = 5.0 "typically" 2112.03221 app. D.1. In its one-example ablation, the CLIP similarity to the prompt is 0.36 with the encoding and 0.26 without it 2112.03221 fig. 5. The paper also shows that CLIP similarity can reward degenerate outputs 2112.03221 §4.2, so this score is weak evidence on its own.

Positional encoding in a radiance field for driving scenes

Neural Scene Graphs passes the position and the view direction through Fourier encodings in its background field 2011.10379 §3.1. Its object fields also take a view direction and a global object position for colour, with the density depending only on the object-frame point and a latent code 2011.10379 §3.2. The paper does not give the number of frequencies for any of these encodings 2011.10379 §3.1. The released code defaults are in the card's Paper vs code section. Card: Neural Scene Graphs for Dynamic Scenes.

The paper's comparisons are between whole methods, not between encodings 2011.10379 §5.2. No result in the card isolates the Fourier encoding.

Box embedding as a condition for object generation

The object denoiser is conditioned on a 3D box [x_c, y_c, z_c, w, l, h, r]. The paper embeds the box with a Fourier embedder, then fuses it with a CLIP text embedding by linear layers 2412.17226 eq. 2 [arxiv:2412.17226 eq. 3]. The box coordinates are normalised to [-1, 1] before the embedding 2412.17226 app. A. The paper takes the embedder from NeRF 2412.17226 §3.2, and does not state its frequencies or width eq. 2. Card: OLiDM: Object-aware LiDAR Diffusion Models for Autonomous Driving.

The paper's object ablation varies the condition, not the embedding. On KITTI-360, the chamfer distance (CD) is 14.3 with no condition, 4.32 with the box only, 9.21 with text only and 1.88 with both 2412.17226 tab. 2. These rows do not isolate the Fourier embedding from the rest of the conditioning path. The paper reports no seeds or standard deviations 2412.17226 §4.1.

When to use which

Implementation sketch

Multi-octave features for a generic input z, as in the diffusion denoiser 2107.00630 §3.4 2107.00630 app. C. Only the frequency list changes between variants.

# Multi-octave Fourier features of the input z  [§3.4] [app. C]
# exponents: integers n, e.g. {7, 8} as written in the paper
def fourier_features(z, exponents):
    channels = [z]                               # keep the raw input
    for n in exponents:
        # frequency 2^n * pi, the paper's notation  [§3.4]
        channels.append(sin((2 ** n) * pi * z))
        channels.append(cos((2 ** n) * pi * z))
    return concat(channels, axis=-1)             # widens the input

The periodic and truncated variants use the same pattern with a different basis. Take the basis from 2307.00379 eq. 22 or eq. 31 rather than from this sketch. The random mapping of Text2Mesh replaces the fixed frequencies with the rows of a Gaussian matrix B 2112.03221 eq. 1; the paper leaves the number of rows open. The box embedder has the same shape, but its frequency list is not given 2412.17226 eq. 2, so you must choose it and record the choice.

Pitfalls

• Frequency convention. The paper writes sin(2^n pi z) for n in {7, 8} 2107.00630 app. C. The released code writes 2 pi 2^n z for n in {6, 7} (see the card's Paper vs code section). These give the same frequencies, so use one convention and check which one you wrote.
• Hard-coded periods. The periodic embedding fixes the period to the domain length. The released Allen-Cahn code uses period 2 on [-1, 1] and 10 frequencies (see the card's Paper vs code section). The paper does not state the frequency count 2307.00379 eq. 22. A different domain needs a new period.
• Extra input for time. The Allen-Cahn code also appends the time coordinate and a constant to the embedding (see the card's Paper vs code section). The paper's eq. 22 has no time input 2307.00379 eq. 22.
• Single-seed ablations. The Allen-Cahn ablation uses one seed, the best of the five in the main table 2307.00379 §3.1.1. Compare the five-seed means instead.
• Schedule interaction. The reported diffusion numbers use learned schedule endpoints. With a fixed schedule the model stays above 4 bits per dimension on the test set 2107.00630 §6.2.
• Text2Mesh scale differs between paper and code. The paper suggests sigma = 5.0 "typically" 2112.03221 app. D.1. The released code defaults to 10.0, and its demos use 5.0 for shoes and 12.0 for person meshes (see the card's Paper vs code section). The scale sets the detail of the result, so record which value you used.
• Text2Mesh ramps the frequencies in by default. This progressive encoding is in the released code but not in the paper. The person demos turn it off (see the card's Paper vs code section).
• Neural-field frequencies are unstated. Neither the position nor the view-direction count is given in the paper 2011.10379 §3.1. The released code defaults to 10 frequencies for position and 4 for view direction and object pose (see the card's Paper vs code section). Choose the counts and record them.
• Box embedder is unspecified. The frequencies, number of bands and output width are not stated 2412.17226 eq. 2. Do not copy the multi-octave list above into a box embedder.
• Box scaling. The paper normalises the box to [-1, 1] before embedding 2412.17226 app. A. Keep that scaling if you follow the paper, because the chosen frequencies act on the scaled values.

Contradictions

None known between papers. The five papers apply Fourier features to different inputs and do not test the same claim, so no reported numbers conflict. Mismatches inside one paper, between its text and its released code, are listed under Pitfalls.

Papers

• Variational Diffusion Models: multi-octave Fourier features on the denoiser input of a diffusion model; reports a likelihood gain on CIFAR-10 and no gain for PixelCNN++.
• Text2Mesh: Text-Driven Neural Stylization for Meshes: random Fourier mapping of vertex positions for a neural style field, scored with CLIP; one-example ablation with and without the encoding.
• Neural Scene Graphs for Dynamic Scenes: Fourier encodings of position and view direction in a background and object radiance fields for driving scenes; frequencies not stated in the paper.
• Residual-based attention in physics-informed neural networks: Fourier embeddings for periodic and Helmholtz PINN problems, with ablations against Dirichlet handling by approximate distance functions.
• OLiDM: Object-aware LiDAR Diffusion Models for Autonomous Driving: Fourier embedding of a 3D object box as part of the condition for a point-cloud diffusion model; the reported ablation varies the condition as a whole.