---
kind: concept
slug: fourier-features
title: Fourier features
summary: >-
  Sine and cosine of scaled inputs, fed to a network next to or in place of its raw input, so it can represent fine detail, periodic structure or a low-dimensional condition. Variants differ in frequencies: several octaves for a diffusion denoiser, a period matched to the domain for PINNs, a random scale for a style field, or a box embedding for objects. Gains come only from each paper's ablations.
tags: [input-encoding, periodicity, diffusion, pinn, conditioning, neural-field]
papers: ["2107.00630", "2112.03221", "2011.10379", "2307.00379", "2412.17226"]
related: [diffusion-models, positional-encoding, implicit-neural-representation, physics-informed-neural-network, neural-radiance-field, neural-style-transfer]
status: draft
updated_at: 2026-10-31
---

## Overview

A network that reads its input directly has to build any fine or periodic
pattern out of smooth functions of that input. Fourier features give it the
sines and cosines of the input at chosen frequencies as extra channels
[arxiv:2107.00630 §3.4] [arxiv:2307.00379 §2.3.3]. The same encoding turns a
low-dimensional condition, such as an object's box, into a richer vector
[arxiv:2412.17226 eq. 2]. It also encodes the coordinates that a neural field
takes as input, such as a mesh vertex or a point and view direction in a
radiance field [arxiv:2112.03221 eq. 1] [arxiv:2011.10379 §3.1]. None of the
cards describes learned frequencies: they are set by hand, drawn from a
Gaussian, or not stated. What changes between variants is the input, the
frequencies and the reason for choosing them.

## Variants

### Multi-octave features on the denoiser input

The denoiser's input is the noisy latent with extra channels
`sin(2^n pi z)` and `cos(2^n pi z)` for a range of integers `n`. The paper
uses `n` in {7, 8} [arxiv:2107.00630 app. C]. Card:
[Variational Diffusion Models](/papers/2107.00630).

The authors report a much better likelihood with the features in their
ablation [arxiv:2107.00630 fig. 5]. The same features gave no benefit when
added to PixelCNN++, and the paper did not observe gains for autoregressive
models in general [arxiv:2107.00630 §6.2] [arxiv:2107.00630 §3.4]. The
reported numbers use learned schedule endpoints together with the features;
with a fixed schedule, the model stays above 4 bits per dimension on the test
set [arxiv:2107.00630 §6.2].

### Period-matched embedding for periodic boundaries

For a PINN with exactly periodic boundary conditions, the input coordinates
are mapped through a Fourier embedding whose period `P` equals the domain
length. The paper gives the one- and two-dimensional forms [arxiv:2307.00379
§2.3.3]; the basis itself is in its eq. 22 and is not reproduced here. Card:
[Residual-based attention in physics-informed neural networks](/papers/2307.00379).

In the paper's Allen-Cahn ablation, the same residual-based attention
weighting gives relative L2 error 5.2e-3 alone and 1.0e-4 with the Fourier
embedding [arxiv:2307.00379 tab. 2]. These are single-seed numbers, which
the paper takes from its best seed [arxiv:2307.00379 §3.1.1].

### Truncated 2D tensor embedding for Helmholtz

For the 2D Helmholtz problem the paper uses a truncated embedding with `m = n
= 5` [arxiv:2307.00379 §3.2] [arxiv:2307.00379 eq. 31]. The paper does not
give the period for this case; the released code fixes it at 2 (see the
card's Paper vs code section). The paper also says the embedding is partly a
comparison, since it contains the solution's trigonometric structure
[arxiv:2307.00379 §3.2.1].

The five-seed means are 8.21e-6 for the Fourier variant and 7.73e-5 for the
approximate-distance-function variant [arxiv:2307.00379 tab. 3]. The two
differ in boundary handling as well: the Fourier variant keeps a boundary
loss, and the other drops it [arxiv:2307.00379 §3.2]. The gap therefore does
not isolate the embedding.

### Random Fourier mapping of vertex positions for a neural style field

Text2Mesh maps each vertex position `p` to `[cos(2 pi B p), sin(2 pi B p)]`,
where the entries of `B` are drawn from a Gaussian with standard deviation
`sigma`. The raw position is kept alongside these features
[arxiv:2112.03221 eq. 1] [arxiv:2112.03221 app. D.1]. The paper gives the
shape of `B` as n × 3 but does not define n, and it gives the encoding a
256-dimensional feature [arxiv:2112.03221 app. D.1]. The features feed a
style field that outputs a colour and a bounded offset along the normal for
each vertex [arxiv:2112.03221 §3.1]. Card: [Text2Mesh](/papers/2112.03221).

The paper calls the frequency scale a hyperparameter that the user controls
[arxiv:2112.03221 §4.1] [arxiv:2112.03221 fig. 7], and suggests `sigma` = 5.0
"typically" [arxiv:2112.03221 app. D.1]. In its one-example ablation, the
CLIP similarity to the prompt is 0.36 with the encoding and 0.26 without it
[arxiv:2112.03221 fig. 5]. The paper also shows that CLIP similarity can
reward degenerate outputs [arxiv:2112.03221 §4.2], so this score is weak
evidence on its own.

### Positional encoding in a radiance field for driving scenes

Neural Scene Graphs passes the position and the view direction through
Fourier encodings in its background field [arxiv:2011.10379 §3.1]. Its object
fields also take a view direction and a global object position for colour,
with the density depending only on the object-frame point and a latent code
[arxiv:2011.10379 §3.2]. The paper does not give the number of frequencies
for any of these encodings [arxiv:2011.10379 §3.1]. The released code
defaults are in the card's Paper vs code section. Card:
[Neural Scene Graphs for Dynamic Scenes](/papers/2011.10379).

The paper's comparisons are between whole methods, not between encodings
[arxiv:2011.10379 §5.2]. No result in the card isolates the Fourier encoding.

### Box embedding as a condition for object generation

The object denoiser is conditioned on a 3D box `[x_c, y_c, z_c, w, l, h, r]`.
The paper embeds the box with a Fourier embedder, then fuses it with a CLIP
text embedding by linear layers [arxiv:2412.17226 eq. 2] [arxiv:2412.17226
eq. 3]. The box coordinates are normalised to [-1, 1] before the embedding
[arxiv:2412.17226 app. A]. The paper takes the embedder from NeRF
[arxiv:2412.17226 §3.2], and does not state its frequencies or width [eq. 2].
Card: [OLiDM: Object-aware LiDAR Diffusion Models for Autonomous Driving](/papers/2412.17226).

The paper's object ablation varies the condition, not the embedding. On
KITTI-360, the chamfer distance (CD) is 14.3 with no condition, 4.32 with the
box only, 9.21 with text only and 1.88 with both [arxiv:2412.17226 tab. 2].
These rows do not isolate the Fourier embedding from the rest of the
conditioning path. The paper reports no seeds or standard deviations
[arxiv:2412.17226 §4.1].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Generative denoiser on images where the likelihood is the target | Multi-octave features on the input [arxiv:2107.00630 §3.4] | The paper reports the likelihood gain [arxiv:2107.00630 fig. 5]; it did not help PixelCNN++ [arxiv:2107.00630 §6.2] |
| Autoregressive likelihood model | Do not expect the same gain from the paper | No improvement observed for autoregressive models [arxiv:2107.00630 §3.4] |
| PINN with periodic boundaries on a known domain | Period-matched embedding [arxiv:2307.00379 §2.3.3] | It makes the periodicity exact by construction; the period must match the domain length |
| PINN with Dirichlet boundaries | Not a Fourier case; the paper uses an approximate distance function instead [arxiv:2307.00379 §2.3.3] | The paper uses it for exact Dirichlet conditions |
| Benchmark where the solution's frequencies are known | Truncated embedding, with care [arxiv:2307.00379 §3.2.1] | The paper says the embedding is partly for comparison, since it contains the solution's frequencies |
| Style field on a fixed mesh, driven by a CLIP loss on renders | Random Fourier mapping, with the scale tuned per object [arxiv:2112.03221 §4.1] [arxiv:2112.03221 fig. 7] | The paper calls the scale a user hyperparameter; its one-example ablation favours having the encoding [arxiv:2112.03221 fig. 5] |
| Radiance field of a driving scene with tracked objects | Fourier encoding of position and view direction, with frequencies you choose [arxiv:2011.10379 §3.1] | The paper names the encoding but not its frequencies, and reports no ablation of it |
| Diffusion generator conditioned on a box, with text, from object-level labels | Fourier-embed the box before fusing it with the text embedding [arxiv:2412.17226 eq. 2] [arxiv:2412.17226 eq. 3] | The paper uses this route. It needs a category, caption and 3D box per object [arxiv:2412.17226 §3.2] [arxiv:2412.17226 app. A], and reports no ablation of the embedder alone [arxiv:2412.17226 tab. 2] |

## Implementation sketch

Multi-octave features for a generic input `z`, as in the diffusion denoiser
[arxiv:2107.00630 §3.4] [arxiv:2107.00630 app. C]. Only the frequency list
changes between variants.

```pseudocode
# Multi-octave Fourier features of the input z  [§3.4] [app. C]
# exponents: integers n, e.g. {7, 8} as written in the paper
def fourier_features(z, exponents):
    channels = [z]                               # keep the raw input
    for n in exponents:
        # frequency 2^n * pi, the paper's notation  [§3.4]
        channels.append(sin((2 ** n) * pi * z))
        channels.append(cos((2 ** n) * pi * z))
    return concat(channels, axis=-1)             # widens the input
```

The periodic and truncated variants use the same pattern with a different
basis. Take the basis from [arxiv:2307.00379 eq. 22] or [eq. 31] rather than
from this sketch. The random mapping of Text2Mesh replaces the fixed
frequencies with the rows of a Gaussian matrix `B` [arxiv:2112.03221 eq. 1];
the paper leaves the number of rows open. The box embedder has the same
shape, but its frequency list is not given [arxiv:2412.17226 eq. 2], so you
must choose it and record the choice.

## Pitfalls

- **Frequency convention.** The paper writes `sin(2^n pi z)` for `n` in {7, 8}
  [arxiv:2107.00630 app. C]. The released code writes `2 pi 2^n z` for `n` in
  {6, 7} (see the card's Paper vs code section). These give the same
  frequencies, so use one convention and check which one you wrote.
- **Hard-coded periods.** The periodic embedding fixes the period to the
  domain length. The released Allen-Cahn code uses period 2 on `[-1, 1]`
  and 10 frequencies (see the card's Paper vs code section). The paper does
  not state the frequency count [arxiv:2307.00379 eq. 22]. A different domain
  needs a new period.
- **Extra input for time.** The Allen-Cahn code also appends the time
  coordinate and a constant to the embedding (see the card's Paper vs code
  section). The paper's eq. 22 has no time input [arxiv:2307.00379 eq. 22].
- **Single-seed ablations.** The Allen-Cahn ablation uses one seed, the best
  of the five in the main table [arxiv:2307.00379 §3.1.1]. Compare the
  five-seed means instead.
- **Schedule interaction.** The reported diffusion numbers use learned
  schedule endpoints. With a fixed schedule the model stays above 4 bits per
  dimension on the test set [arxiv:2107.00630 §6.2].
- **Text2Mesh scale differs between paper and code.** The paper suggests
  `sigma` = 5.0 "typically" [arxiv:2112.03221 app. D.1]. The released code
  defaults to 10.0, and its demos use 5.0 for shoes and 12.0 for person
  meshes (see the card's Paper vs code section). The scale sets the detail
  of the result, so record which value you used.
- **Text2Mesh ramps the frequencies in by default.** This progressive
  encoding is in the released code but not in the paper. The person demos
  turn it off (see the card's Paper vs code section).
- **Neural-field frequencies are unstated.** Neither the position nor the
  view-direction count is given in the paper [arxiv:2011.10379 §3.1]. The
  released code defaults to 10 frequencies for position and 4 for view
  direction and object pose (see the card's Paper vs code section). Choose
  the counts and record them.
- **Box embedder is unspecified.** The frequencies, number of bands and output
  width are not stated [arxiv:2412.17226 eq. 2]. Do not copy the multi-octave
  list above into a box embedder.
- **Box scaling.** The paper normalises the box to [-1, 1] before embedding
  [arxiv:2412.17226 app. A]. Keep that scaling if you follow the paper, because
  the chosen frequencies act on the scaled values.

## Contradictions

None known between papers. The five papers apply Fourier features to
different inputs and do not test the same claim, so no reported numbers
conflict. Mismatches inside one paper, between its text and its released
code, are listed under Pitfalls.

## Papers

- [Variational Diffusion Models](/papers/2107.00630): multi-octave Fourier features on the denoiser input of a diffusion model; reports a likelihood gain on CIFAR-10 and no gain for PixelCNN++.
- [Text2Mesh: Text-Driven Neural Stylization for Meshes](/papers/2112.03221): random Fourier mapping of vertex positions for a neural style field, scored with CLIP; one-example ablation with and without the encoding.
- [Neural Scene Graphs for Dynamic Scenes](/papers/2011.10379): Fourier encodings of position and view direction in a background and object radiance fields for driving scenes; frequencies not stated in the paper.
- [Residual-based attention in physics-informed neural networks](/papers/2307.00379): Fourier embeddings for periodic and Helmholtz PINN problems, with ablations against Dirichlet handling by approximate distance functions.
- [OLiDM: Object-aware LiDAR Diffusion Models for Autonomous Driving](/papers/2412.17226): Fourier embedding of a 3D object box as part of the condition for a point-cloud diffusion model; the reported ablation varies the condition as a whole.