Implicit neural representation (INR)

Overview

An implicit neural representation (INR) is a multilayer perceptron that takes a coordinate, such as a pixel position or a 3D point, and returns the signal value there. The signal is stored in the network weights rather than in a grid of samples 2301.05187 §3.1. Each hidden layer applies a linear map and a nonlinearity σ 2301.05187 eq. 1. The choice of σ largely decides which frequencies the network fits first and how readily it also fits noise 2301.05187 §1 2301.05187 §3.2.

Fitting is done by gradient descent on a loss between the forward-modelled output and the measurements, with Adam and an l2 loss and no other regularisation in the WIRE paper 2301.05187 §4.

A second design axis is the training signal. Where WIRE fits a fixed set of measurements, iSDF fits a signed distance field (SDF) online, as posed depth frames arrive, and it constrains each prediction with a bound taken from the measured surface 2204.02296 §4 2204.02296 §4.3. Its MLP uses a positional embedding and softplus layers rather than a wavelet 2204.02296 §4.1.

A third axis is conditioning. NeuSE uses one occupancy decoder for every object of a category. The decoder takes a query point and a per-object latent code, so the object's identity enters through the code rather than through a separate set of weights 2303.07308 eq. 3 2303.07308 §3.1.

A fourth axis is the coordinate frame and the coordinate itself. Neural Scene Graphs gives each dynamic object its own field in that object's frame, so objects can be moved or removed by editing the graph 2011.10379 §3. StyleGAN-V treats a video as a function of continuous time and generates a frame at any real-valued time 2112.14683 §3.1.

Two further axes appear in the newer cards. NDF trains an occupancy network and then reads the activations of all its layers as a descriptor for every point, which is used to match poses rather than to reconstruct a signal 2112.05124 §II-A 2112.05124 eq. 4. DONeRF keeps a coordinate MLP for radiance but gives the sample placement to a second network, so the expensive network is evaluated at few points per ray 2103.03231 §5.1.

Variants

Eight variants have paper cards. WIRE and iSDF each fit one signal, and they differ in the activation (WIRE) and the supervision (iSDF). NeuSE shares one decoder across objects of a category and differs in its conditioning. GeFF goes further: one feed-forward encoder produces the latent for a whole scene from posed RGB-D frames, so a new scene needs no per-scene fitting. Neural Scene Graphs builds a graph of fields for one driving scene, and StyleGAN-V uses time as its coordinate for video. NDF uses a shape-conditioned descriptor field for manipulation, and DONeRF uses a depth oracle to place the samples of a radiance field. The other activations that the WIRE paper compares against (sine, Gaussian, ReLU with positional encoding, and a Gabor-filter multiplicative network) are named as baselines there, but they have no card yet 2301.05187 §4.

Wavelet activation (WIRE)

The nonlinearity is a complex Gabor wavelet with frequency ω0 and width s0. Its complex exponential gives frequency selectivity, as in a sine activation, and its Gaussian window gives spatial compactness, as in a Gaussian activation 2301.05187 §3.2 2301.05187 eq. 2. Because the window already keeps each layer local, the default uniform random initialisation works for any ω0 and s0 2301.05187 §3.2 2301.05187 app. A.1.

The paper also gives a real-valued form, sin(ω0 x) times a Gaussian window. Setting s0 to 0 recovers a sine activation, and setting ω0 to 0 recovers a Gaussian one 2301.05187 §3.4. For two or more spatial dimensions, the paper adds one Gaussian window per extra coordinate to the same layer 2301.05187 §3.5 2301.05187 eq. 5. Card: WIRE: Wavelet Implicit Neural Representations.

Online signed distance field (iSDF)

The INR maps a 3D point to a signed distance, with the sign positive in front of the surface 2204.02296 §3 2204.02296 §4. It is a randomly initialised MLP with no pretraining and no ground-truth SDF [arxiv:2204.02296 §1] 2204.02296 §4. Each iteration samples points along rays from a few posed depth frames: the two latest frames and three keyframes 2204.02296 §4.2.

The supervision is the distinctive part. For each sampled point, the bound is the signed distance to the nearest measured surface point among all rays in the batch 2204.02296 eq. 4. Points in free space take a one-sided loss that is zero between 0 and the bound 2204.02296 eq. 6. Points near the surface take an L1 term to the bound 2204.02296 eq. 7 2204.02296 eq. 8. Cosine and Eikonal terms regularise the gradient 2204.02296 eq. 9 2204.02296 eq. 10. The paper argues this avoids the voxel grid and surface-fusion step of older real-time methods 2204.02296 §1 2204.02296 §2.

The paper assumes camera poses from an external tracker and does not do full SLAM 2204.02296 §4. Card: iSDF: Real-Time Neural Signed Distance Fields for Robot Perception.

Latent-conditioned occupancy decoder (NeuSE)

The decoder Φ maps a query point x and a latent code z to an occupancy value in [0, 1] 2303.07308 eq. 3 2303.07308 §3.1. The code is not a weight set. It is a latent point cloud of k = 512 points 2303.07308 §5.1 2303.07308 app. A1, produced by an encoder from a partial point cloud of one object. The encoder is built so that the code transforms with the object: rotating, translating or scaling the input transforms the code in the same way 2303.07308 eq. 1 2303.07308 eq. 2 2303.07308 eq. 4.

That equivariance is what makes the codes useful for pose. Two codes of the same object give a relative SE(3) transform in closed form, by Horn's method, without rendering the object and fitting it 2303.07308 §3.3 2303.07308 eq. 6. Ambiguous objects, such as upright bottles, are trained with a Chamfer-distance loss on the aligned clouds, not a single ground-truth pose 2303.07308 §3.4 2303.07308 eq. 8.

The decoder is trained for each category separately, on simulated ShapeNet objects rendered in PyBullet 2303.07308 §5.1 2303.07308 app. A1. Card: NeuSE: Neural SE(3)-Equivariant Embedding for Long-Term Object-Based Simultaneous Localization and Mapping.

Generalizable scene feature field (GeFF)

GeFF is a scene-level, generalizable NeRF. A trainable encoder reads N posed RGB-D frames and outputs N latent vectors, with no optimisation for the new scene 2403.07563 §3.1 2403.07563 §3.2. Two decoders then read any 3D point together with the latents. One gives a signed distance, used for geometry and opacity along rays. The other gives a feature vector, distilled from a frozen 2D vision-language teacher, which is used for open-vocabulary queries 2403.07563 §3.2 2403.07563 eq. 2 [arxiv:2403.07563 eq. 4]. The decoders are isotropic, so they do not take the viewing direction 2403.07563 §3.2.

The training signal combines two sources. Novel views with ground truth supervise the rendered depth and features, and the SDF is trained with the iSDF loss plus an Eikonal term 2403.07563 §3.2 2403.07563 eq. 6. The paper trains the feature head to match the teacher's features on those novel views, not on the input views the encoder sees 2403.07563 §3.2. Pre-training is on ScanNet for 50 epochs; the paper reports six days on eight GPUs 2403.07563 §4.1. Card: Learning Generalizable Feature Fields for Mobile Manipulation.

Where the NeuSE latent encodes one object, the GeFF latents cover a whole scene. Both let one decoder read a latent at arbitrary points. NeuSE's encoder is designed to be SE(3)-equivariant 2303.07308 eq. 1, and the GeFF paper does not make an equivariance claim for its latents [arxiv:2403.07563 §3.1].

Object-graph radiance fields (Neural Scene Graphs)

The scene is a directed acyclic graph with a camera, one static background field and one radiance field per dynamic object 2011.10379 eq. 1 2011.10379 §3. Each object field is defined in its own frame. A world point is mapped into that frame using the object's box size, so points inside the box fall in [-1, 1] 2011.10379 §3.2 2011.10379 eq. 8. Objects of one class share weights and differ through a 256-dimensional latent vector 2011.10379 §3.2 2011.10379 eq. 6.

The object density depends only on the object-frame point and its latent. The colour also depends on the view direction and on the object's global position, which keeps the density consistent across poses 2011.10379 §3.2 2011.10379 eq. 7 2011.10379 eq. 9. The background is sampled on a few planes parallel to the camera image plane, not ray-marched through a volume 2011.10379 §4.1. Edges between nodes carry rigid transforms or scales, so moving or removing an object is an edit to the graph 2011.10379 §3. The training data are driving videos plus object tracks, which come from manual labels or from a tracker 2011.10379 §5.1. Card: Neural Scene Graphs for Dynamic Scenes.

Time-coordinate video generator (StyleGAN-V)

StyleGAN-V is a GAN, not a per-signal fit, but it uses the coordinate idea along the time axis. A video is modelled as x(t) for continuous t ≥ 0, and training uses sparse subsamples of it 2112.14683 §3. A content code sets the look of a clip and is shared by all its frames [arxiv:2112.14683 §3.1]. A motion code is computed for any query time from a grid of noise codes, using wave-based time embeddings that are acyclic and can be queried at real-valued times 2112.14683 §3.1 2112.14683 eq. 4. The frames are generated independently, so generation is not autoregressive 2112.14683 §3.1. The synthesis network is unchanged except for its constant input, which takes the motion code 2112.14683 fig. 5. Card: StyleGAN-V.

Descriptor field for pose matching (NDF)

NDF's core network is an occupancy decoder conditioned on one object's point cloud. A point's descriptor is the concatenation of the activations of every layer of that network 2112.05124 eq. 4 2112.05124 §II-A. The encoder and the decoder are made SE(3)-equivariant with Vector Neurons, so transforming the cloud and the query point by the same rigid motion moves the descriptor field with them 2112.05124 eq. 5 2112.05124 eq. 7. Training is 3D reconstruction from partial clouds, with no keypoint or correspondence labels 2112.05124 §II-A.

A rigid body, such as a gripper or a rack, is represented by query points placed in its box, and its pose descriptor stacks the descriptors of those points 2112.05124 §II-B 2112.05124 eq. 10. In the few-shot pick-and-place procedure, the pick and relative-place descriptors are averaged over the demonstrations. A test pose is found by minimising the L1 distance between the candidate's pose descriptor and that average 2112.05124 §II-C 2112.05124 eq. 11. Card: Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation.

Oracle-guided sampling of a radiance field (DONeRF)

DONeRF keeps a coordinate MLP that returns colour and opacity, but it changes how the samples on each ray are chosen. A depth oracle is evaluated once per ray. It scores depth classes on a logarithmic axis, and the scores are read as a piecewise-constant density whose inverse CDF sets the sample depths 2103.03231 §5.1 2103.03231 §5.2. A shading MLP then evaluates only those few samples 2103.03231 §3.

The oracle is trained as a classifier against the depth of the first surface, so training needs a reference depth for each pixel 2103.03231 §5.1 2103.03231 §6.1. Its targets are spread across neighbouring rays and depth classes, which makes the oracle keep a little extra empty space 2103.03231 eq. 7 2103.03231 eq. 8 2103.03231 §5.1. Without a reference depth, the paper takes depth maps from a dense NeRF instead 2103.03231 §6.1. Card: DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks.

When to use which

The papers' results are author-reported. Each row names the constraint and the evidence for it.

Caveats for the rows above:

• Most inverse-problem results use one image or one volume, so generalisation to other signals is not shown 2301.05187 §4.2 2301.05187 app. A.4.
• The implicit-bias argument rests on an empirical NTK study on 64 by 64 images, and the authors argue rather than prove it 2301.05187 §3.3.
• Settings were picked to give the fastest approximation on the test signal, with no validation split stated 2301.05187 §4.1.
• The iSDF evaluation uses simulated ReplicaCAD and handheld ScanNet sequences, and the paper describes no robot-platform experiment 2204.02296 §5.1.
• The iSDF results are read from text and a figure. The authors say the baselines are plotted in figure 8 only 2204.02296 §6.2.
• The iSDF timing is for one optimisation iteration on one hardware setup. Frame rate and tracking are not evaluated 2204.02296 §5.2 2204.02296 §6.3.
• The released iSDF configs use a different bound from the one the paper uses in its experiments. The iSDF card lists the differences between paper and code.
• NeuSE is trained only in simulation, on five object categories 2303.07308 §5.1. Its real-world tests add two objects of an untrained size, and the paper does not treat those runs as comparable to the controlled sequences 2303.07308 §7.2.
• The NeuSE timings (6 ms of network inference per object, about 4 fps for the full localisation system in the crowded controlled setting) are on the authors' laptop, with sequential processing 2303.07308 §7.
• GeFF's main comparison mixes timings. LERF and ConceptGraphs were run offline with all frames batched, and ConceptGraph-Online ran on the robot with dropped frames 2403.07563 §4.1. The 62.4% versus 43.2% gap therefore compares different operating modes as well as different methods.
• GeFF's robot results come from one platform in four rooms, with 3 trials per object in the scene-change cells and no confidence intervals [arxiv:2403.07563 §4.1] 2403.07563 §4.2 A4.
• Neural Scene Graphs reports numbers only for three KITTI scenes. The qualitative results, the tracker comparison and the 3D detection are described in words, with no figures in the text 2011.10379 §5.2 2011.10379 fig. 9.
• Neural Scene Graphs uses a different box scale for some object labels and a different latent penalty from the paper's formula. The Neural Scene Graphs card lists these differences.
• StyleGAN-V's FVD numbers come from the authors' own re-trained baselines and FVD protocol, so they are not comparable with numbers from other papers 2112.14683 §4.1 2112.14683 app. C.
• StyleGAN-V's frame count is a setting. Table 3 has k = 2 with a lower SkyTimelapse FVD16 than the default k = 3, while k = 3 is lowest on FaceForensics 2112.14683 tab. 3.
• NDF's numbers are simulation results in PyBullet, with 10 demonstrations per task and one model shared by mug, bowl and bottle 2112.05124 §III. The authors' released code differs from the paper in several places, such as descriptor normalisation and input scale. The NDF card lists them.
• DONeRF's baseline NeRF is a PyTorch port with the same layout as DONeRF's shading network, not the original NeRF network, and it uses a different batch size 2103.03231 app. E 2103.03231 §6.1. The comparison matches capacity, not the full training recipe.
• The DONeRF official code is non-commercial, so the DONeRF card is written from the paper alone and no code details are used here.

Implementation sketch

The WIRE activation and its forward pass, following 2301.05187 §3.1 and 2301.05187 §3.2. Complex arithmetic keeps phase through the hidden layers, and only the real part of the output is the signal.

# Gabor activation, complex-valued  [arxiv:2301.05187 eq. 2]
# Input:  pre-activation x (N, H)
# Output: activation (N, H), complex

def gabor(x, omega_0, scale_0):
    # e^{j w0 x} times the Gaussian window e^{-|s0 x|^2}
    return exp(1j * omega_0 * x - abs(scale_0 * x) ** 2)

# Forward pass of a WIRE-style INR  [arxiv:2301.05187 §3.1] [§3.2]
# Input:  coords (N, D) in [-1, 1]
# Output: signal (N, C), real

def inr_forward(coords, first, hidden, readout, omega_0, scale_0):
    # First layer: real weights, output is complex  [§3.2]
    h = gabor(first(coords), omega_0, scale_0)
    # Hidden layers: complex weights and activations  [eq. 1]
    for layer in hidden:
        h = gabor(layer(h), omega_0, scale_0)
    # Read out and keep the real part as the signal  [§3.2]
    return real(readout(h))

Starting values the paper gives for images: ω0 = 20 and s0 = 10 for WIRE, and three hidden layers 2301.05187 §4.1. The learning rate is not stated for images. The authors' code sets some values differently from the paper; see the WIRE card.

The batch bound used by iSDF for one sampled point, following 2204.02296 §4.3 and 2204.02296 eq. 4. The sign is positive in front of the surface.

# Batch bound for one sampled point  [arxiv:2204.02296 eq. 4]
# Input:  sample x (3,), measured surface points P (K, 3), sign s in {-1, 1}
# Output: scalar bound on the predicted signed distance

def batch_bound(x, P, s):
    # Distance from x to the nearest measured surface point in the batch
    return s * min(norm(x - p) for p in P)

The paper gives this as the bound for the free-space loss, zero for a prediction between 0 and the bound 2204.02296 eq. 6. The truncation distance is not stated in the paper, so an implementer must choose it 2204.02296 §4.4.

The NeuSE decoder query, following 2303.07308 eq. 3 and 2303.07308 eq. 5. The latent is centred, and the query is shifted by the latent's centre so that the decoder sees a translation-free input.

# Occupancy query of a latent-conditioned decoder  [arxiv:2303.07308 eq. 5]
# Input:  query points x (M, 3), latent z (k, 3) from the encoder
# Output: occupancy in [0, 1], (M,)

def occupancy(x, z, decoder):
    # Centre of the latent code  [eq. 5]
    z_bar = mean(z, axis=0)                            # (3,)
    # Decoder sees the centred latent and the shifted query  [eq. 5]
    # The decoder ends in a sigmoid  [arxiv:2303.07308 tab. A2]
    return decoder(x - z_bar, z - z_bar)               # (M,)

Contradictions

None across papers: the eight cards use different activations, supervision, conditioning and domains, and no paper compares directly against another of the eight. Within WIRE and iSDF, the authors' released code differs from the paper for some settings. The WIRE card and the iSDF card list the differences. NeuSE and GeFF have no public code, so their cards record no code differences.

Within GeFF, the card's sources disagree on the teacher. The method names MaskCLIP as the teacher 2403.07563 §3.2, while the training details name a ViT-L CLIP model 2403.07563 §4.1. The paper does not say whether MaskCLIP is applied to a ViT-L CLIP backbone, so both are reported here.

Within NeuSE, the text and the table disagree on one reduction. The text gives a 65% ATE improvement on part of one trajectory 2303.07308 §7.1. Table 3 gives full-trajectory reductions against the point-cloud baseline of about 58% (planar) and 66% (non-planar) 2303.07308 tab. 3. Both figures are stated here as the paper gives them, without choosing between them.

Within StyleGAN-V, the text and Table 1 disagree on how far the method leads. The text says it "significantly outperforms" the others on almost all benchmarks 2112.14683 §4.1. Table 1 supports the lowest FVD16 on three of four benchmarks, and VideoGPT is lowest on RainbowJelly [arxiv:2112.14683 tab. 1]. The abstract's average-improvement figure is not shown in the sections read here 2112.14683 abstract. Both sides are stated as the paper gives them.

Within DONeRF, the paper gives four speed-up ranges for the FLOP reduction against NeRF: up to 48 times in the abstract 2103.03231 abstract, 24 to 98 times in the introduction 2103.03231 §1, 15 to 78 times in §6.3 2103.03231 §6.3, and 48 to 78 times in the caption of tab. 2 2103.03231 tab. 2. All four are stated here as the paper gives them.

Within NDF, the introduction says overall success is above 85% 2112.05124 §I. Table I gives 0.87 to 0.91 for upright objects, which fits that claim, and 0.58 to 0.78 for arbitrary poses, which does not 2112.05124 tab. I.

Papers

• WIRE: Wavelet Implicit Neural Representations: a complex Gabor wavelet as the INR activation, with results on images, occupancy, denoising, super-resolution, CT and NeRF.
• iSDF: Real-Time Neural Signed Distance Fields for Robot Perception: an MLP trained online from posed depth images, supervised by a batch bound from the nearest measured surface point.
• NeuSE: Neural SE(3)-Equivariant Embedding for Long-Term Object-Based Simultaneous Localization and Mapping: a per-category occupancy decoder conditioned on an SE(3)-equivariant latent point cloud, used for object association, pose constraints and change detection in SLAM.
• Learning Generalizable Feature Fields for Mobile Manipulation: a feed-forward scene encoder with SDF and distilled language-feature decoders, used for navigation and part-level grasping on a legged robot.
• Neural Scene Graphs for Dynamic Scenes: one static background field and one field per tracked object in a scene graph, with class-shared weights and per-object latents, so objects can be edited and re-rendered.
• StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2: a StyleGAN2-based video GAN whose motion codes are a continuous function of time, so frames are generated at any real-valued time.
• Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation: an equivariant descriptor field from occupancy-network activations, used to match demonstrated poses for few-shot pick-and-place.
• DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks: a depth oracle that places a few samples per ray for a shading MLP, reaching NeRF-level PSNR with few evaluations.