Modulated Periodic Activations for Generalizable Local Functional Representations

In plain words

Many signals, such as photos, video frames, 3D shapes or lighting setups, can be stored as a small neural network that answers "what colour or distance is at this position?". Usually one network is trained for each signal, which is slow and does not carry over to new signals. This paper trains one shared model that can represent many signals. Each signal gets a short list of numbers, its latent code, and a second small network turns that code into instructions that shape the main network.

The main network uses wave-like (sine) activation functions, which capture fine detail well. The second network, which reads the code, controls how strong each layer of the main network is and how it bends. Feeding the code in as an extra input works poorly with waves, so the code is used as this control signal instead.

To handle large images, the signal is cut into a grid of overlapping tiles. Each tile gets its own code, and the tiles are blended at the seams. The codes can be produced by an encoder network, or they can be found by optimising them against a fixed model. The result is a compact, continuous representation that can be queried at any position, including between the original pixels.

TL;DR

A synthesis MLP with sine activations is gated layer by layer by a ReLU modulator that reads a per-signal latent code, so one trained model represents many images, videos or shapes. Tiling the signal into overlapping local patches, each with its own code, scales the method to high-resolution inputs. Author-reported results beat the compared MLP baselines on CelebA, CIFAR-10, ShapeNet cars, Vimeo-90k and Div2K, and encode high-resolution images in about 15 s instead of about 60 min per image. Released code exists but has no licence, so use it to check details, not to copy text.

Problem

Coordinate MLPs (implicit or functional representations) are usually fitted to one signal at a time. Generalising them to many signals costs quality: the paper reports that its baselines degrade or fail to converge on varied data §4.1 tab. 2. Two obstacles are named. ReLU MLPs miss high-frequency detail, and sine networks do not accept a latent code in the obvious way. Concatenating the code to the coordinates only shifts the phase of the first layer, which the paper calls a severe expressivity limit §3.2 eq. 6 fig. 4. Global fits of large images also need costly models, because a full image has a broad frequency spectrum §1 fig. 2.

Method

The model has two parts, a synthesis network and a modulation network, both defined in §3.1.

Synthesis network §3.1.1. Coordinates x in the input domain (n = 2 for images) pass through K hidden layers. Each layer multiplies a sine of an affine map of the previous hidden features by a modulation vector alpha_i. The output layer is linear to m = 3 colour channels eq. 1 eq. 2.

Modulation network §3.1.2. A ReLU MLP reads the latent code z (d = 256 in the paper §3.1) and produces one modulation vector per hidden layer. The paper feeds z into every layer as a skip connection eq. 3 eq. 4. Because alpha_i scales the amplitude of each sine layer, z also changes the frequency and phase of later layers, since the modulated outputs feed the next sine §3.1.2 eq. 5.

Expressivity §3.2. Concatenating z with x before the first sine layer eq. 6 changes only the phase of that layer. The paper's modulator instead controls amplitude, frequency and phase at every layer fig. 4.

Local functional representation §3.3. The domain is split into a regular grid of tiles, and each tile gets its own latent code z_i. The tile function receives normalised local coordinates in [0, 1]^n. Tiles overlap, and the contributions of overlapping tiles are weighted linearly by distance to tile centres §3.3 fig. 5. The paper says the weighting is n-linear; it does not state the overlap ratio.

Training §3.4.

• Auto-encoder §3.4.1: a convolutional encoder predicts z from the discrete signal; encoder, modulator and synthesis are trained together §4.1.
• Auto-decoder §3.4.2: the codes for all training signals are optimised together with the network weights, as in DeepSDF. A new signal is fitted by optimising a fresh code with the weights frozen, using the same objective. Codes start from N(0, s²) with s = 10^-2 §3.4.2.

Pseudocode for one modulated evaluation. The paper gives no frequency scale, so omega is a placeholder (see Hyperparameters and Paper vs code).

# Modulated synthesis for one signal or tile  [§3.1] [eq. 2] [eq. 3-4]
# Input:  coords (N, 2) in [0, 1]^2; latent (256,)
# Output: rgb (N, 3)

def modulated_synthesis(coords, latent, params):
    # Modulator: ReLU MLP on the latent, one gain vector per hidden layer
    # [§3.1.2] [eq. 3] [eq. 4]
    gains = modulator(latent, params.modulator)  # K vectors, each (W,)
    # First sine layer; the paper gives no frequency scale, so omega is
    # a free choice here [§3.1.1] [eq. 2]
    hidden = sin(omega * (coords @ w_first.T + b_first))  # (N, W)
    for i in range(K):
        # Gate each hidden layer: alpha_i * sin(W_i h + b_i)  [§3.1.1] [eq. 2]
        pre = hidden @ w[i].T + b[i]  # (N, W)
        hidden = gains[i] * sin(pre)  # (N, W)
    # Linear read-out to colour  [§3.1.1] [eq. 1]
    return hidden @ w_out.T + b_out  # (N, 3)

Local evaluation at a query point. The paper states the blending in words §3.3 fig. 5; the exact weights are an implementation choice.

# Local evaluation at one query point  [§3.3] [fig. 5]
# Input:  x (2,) in the signal domain; codebook (T, 256), one code per tile

def local_value(x, codebook):
    # Find the overlapping tiles whose support contains x  [§3.3]
    tiles = covering_tiles(x)
    total = zeros(3)
    for tile in tiles:
        # Normalised coordinates inside this tile  [§3.3]
        local = (x - tile.origin) / tile.size  # (2,)
        # Linear weight from the distance to the tile centre  [§3.3]
        weight = linear_weight(x, tile.centre)  # scalar
        # Tile function with this tile's code  [§3.3] [eq. 2-4]
        value = modulated_synthesis(local, codebook[tile.id])
        total = total + weight * value
    # Normalisation of the weights is not stated in the paper
    return total

Test-time fitting in the auto-decoder setting. The paper states the initialisation and the objective, but not the optimiser, learning rate or step count.

# Auto-decoder fitting of a new signal  [§3.4.2]
# Network weights are frozen; only the latent code is optimised.

def fit_latent(coords, targets, params, steps):
    # Random start, N(0, s^2) with s = 1e-2  [§3.4.2]
    latent = normal(0.0, 1e-2, size=(256,))
    for step in range(steps):
        # Same objective as in training, with weights fixed  [§3.4.2] [eq. 7]
        output = modulated_synthesis(coords, latent, params)
        loss = mean((output - targets) ** 2)  # (scalar)
        # Optimiser and learning rate are not stated in the paper
        latent = latent - learning_rate * grad(loss, latent)
    return latent

Hyperparameters

Values are as the paper states them unless the Where column names the code. "Not stated" means the text read did not give a value.

Results

Author-reported. The paper does not state the hardware in the text read. Baselines are ReLU-MLP, FFN, SIREN+ (sine network with the latent concatenated to coordinates), and HyperNet-SIREN §4. Each table is cited in the Source column.

Global generalisation, CelebA (32 x 32 training images, 1x and 2x PSNR) tab. 1:

Global generalisation, CIFAR-10 (10 K test images; PSNR 1x and 2x) tab. 2:

ShapeNet cars, auto-decoder, bi-directional Chamfer distance (3514 shapes) tab. 3:

Video, Vimeo-90k (7 K test videos, PSNR) tab. 4:

Local, Div2K (50 unseen 2K images, 32 x 32 tiles; PSNR 1x) tab. 5. Global methods are overfitted per image; the local methods are pre-trained:

3D scenes, ThreedScans, Chamfer distance x 10^-5 tab. 6:

Videos, high resolution (6 Pexels clips at 640 x 256; PSNR) tab. 7:

The text says this is about 1000x faster §4.2; the table times give roughly 900x. Relighting is shown qualitatively only §4.3 fig. 11. Qualitative results are also shown for CelebA fig. 7, cars fig. 8, video cuts fig. 9 and 3D scenes fig. 10.

Paper vs code

Official code: https://github.com/ishit/modsiren, cloned at commit 6108cc1fafb8eea2dda2891e029e483944f3a886. The paper does not link it. The paper has no project page link, and the arXiv metadata has no comments field. GitHub search on the title phrases and method phrases returned nothing. The repository was found through the first author's account (ishit, profile name Ishit Mehta). Its folder and script names match the paper's experiments (CelebA, CIFAR, Div2K, video, shapes, relighting). The repository has no README and no LICENSE, so it is cited only, with no verbatim snippets. Attribution rests on the owner check above and should be confirmed by a human before the card is reviewed.

Discrepancies between paper and code. Each item cites both sides.

• Layer gating indices. Paper eq. 3 defines the modulator's first output h'_0 but gives alpha_{i+1} = h'_{i+1}, so alpha_1 is never defined eq. 3 eq. 4. The code applies a sine layer with no gate first, then gates the next K sine layers models_bak.py.
• Modulator skip. Paper feeds z into every modulator layer by concatenation ([h_i z]) eq. 4. The code adds a skip of the first-layer embedding after each ReLU layer models_bak.py. The cleaner ModSiren class in the release does not re-feed z at all, and gates only the first K-1 hidden layers models.py.
• Frequency. Paper eq. 2 has no frequency scale eq. 2. The code uses omega = 30 in hidden layers, and the first layer uses freq_scale (1.0 in the image configs). Init is uniform in +/- sqrt(6/n)/omega models_bak.py models_bak.py modsiren.yml.
• Modulator input. Paper says the modulator acts on z alone §3.1.2. The single-image config sets concat: True, which also feeds coordinates into the modulator modsiren.yml models_bak.py.
• Latent init. Paper uses s = 1e-2 §3.4.2. Image configs use 1e-3 modsiren.yml and shape configs 1e-4 modsiren.yml.
• Epochs. Paper says all models train for 1000 epochs §4.1. The CIFAR config uses 400 modsiren.yml.
• Image loss. Paper eq. 7 is a squared L2 norm with no regulariser eq. 7. The code takes the mean of the squared error over elements and adds reg_lam times mean z^2 train_img_gen.py. The relative weight of the regulariser depends on that normalisation, and the paper does not say which one it uses.
• Shape loss. Paper uses an L1 penalty §4.1. The code clamps both prediction and ground-truth SDF to +/-0.1 before the L1 loss train_shape_gen.py train_shape_gen.py modsiren.yml.
• Config names. config/img_gen/celeba/modsiren_32.yml points to a castle image, not CelebA modsiren_32.yml. The mapping from config files to the table rows is therefore inferred, not stated.
• Two model files. The image scripts import LocalMLP from models_bak.py train_img_gen.py, not from models.py. Results should be traced to models_bak.py.

Not compared: the toy concatenation experiment (concat_expriment/main.py), the relighting and video scripts, and the encoder architectures. The tile-blending weights are compared only in Pitfalls, because the paper leaves them unspecified.

Pitfalls

• Gate indexing is ambiguous (see Paper vs code). An implementer must choose whether the first sine layer is gated, and whether the modulator emits K or K+1 vectors. The code gates K layers after an ungated first layer.
• The frequency scale is not in the paper. The code's omega = 30 with SIREN-style init is a large effective frequency. A reimplementation following eq. 2 literally, with no omega and default init, will produce different detail.
• Test-time fitting is under-specified. The optimiser, learning rate and step count are not stated. Only the initial code scale is given §3.4.2. The released test-time routine uses Adam with a separate val_lr setting, which is not in the image configs read here train_img_gen.py modsiren.yml.
• Tile overlap and blend weights are not given. The code uses half-cell offsets and a linear ramp over half a cell, applied along rows and then columns train_single_img_patches.py. Normalisation is not stated.
• Continuity depends sharply on the FFN scale. With sigma = 10, FFN's 2x PSNR falls to 4.22 on CelebA tab. 1 and to 5.75 on CIFAR-10 tab. 2.
• Dataset counts disagree. The text gives CelebA as 167 K train and 33 K test §4.1, but the Table 1 caption says 200 K images.
• CIFAR-10 2x evaluation disagrees. The text says ground truth is resampled bicubically §4.1; the Table 2 caption says bilinear upsampling.
• Costs are hidden in the comparisons. Global baselines are overfitted per image, which takes about 60 min per Div2K image and about 15 h per video tab. 5 tab. 7. The local method's speed depends on a pre-trained model on Vimeo-90k §4.2.
• HyperNet-SIREN needs about 72 M parameters against 1.3 M for this method tab. 1.
• SIREN+ does not converge on CIFAR-10 or Vimeo-90k in the authors' runs tab. 2 tab. 4. The paper attributes this to data variability, which it does not test.
• The Div2K single-instance results use 2K images with 32 x 32 tiles and about 2 M tiles in training §4.2.
• The released repository has no README and no LICENSE, so reuse terms are unknown. Treat it as reference material only.

Lineage

• Builds on: sine-activated networks trained for signal fitting (SIREN, Sitzmann et al. 2020), for the sine layers and initialisation §1 models_bak.py. DeepSDF (Park et al. 2019) for the auto-decoder setup §3.4.2. Local shape latents (Chabra et al. 2020; Jiang et al. 2020) for the tiled representation §1. Hypernetworks (Ha et al. 2016) are a baseline §4.
• Compared against: ReLU-MLP, FFN (Tancik et al. 2020), SIREN+, HyperNet-SIREN §4 fig. 6. DeepSDF and DeepSDF with SIREN or FFN on ShapeNet cars tab. 3. Stated as baselines; no arXiv ids are given in the paper, so none are linked here.
• Superseded by: not known from content already in this wiki.