Splatter Image: Ultra-Fast Single-View 3D Reconstruction

In plain words

Many methods that rebuild a 3D object from photos fit a model to that object slowly, one scene at a time. Splatter Image takes a shortcut. A standard image-to-image neural network reads one photo and outputs a grid with the same width and height as the photo. Each cell of that grid stores one small 3D blob, a Gaussian, that sits roughly where the object's surface is. The blobs can be drawn onto a new viewpoint very quickly with Gaussian Splatting, a fast rendering technique. Training compares rendered views with real photos, so the network can also be judged by perceptual image metrics. Because the output is a grid of blobs, the network can fill in parts of the object the camera never saw, and it can take several photos at once by merging their blobs.

TL;DR

Splatter Image maps one image to one 3D Gaussian per pixel with a 2D U-Net and renders them with Gaussian Splatting §3.2 §3.1. Training minimises an L2 loss on rendered target views, with an optional LPIPS term added in a second stage eq. 4 app. C. Several input views are handled by merging their Gaussians after a pose change, with cross-view attention in the U-Net §3.4 §3.6. Author-reported: 24.00 PSNR on ShapeNet-SRN cars and 21.06 PSNR on Google Scanned Objects against OpenLRM's 18.06 tab. 1 tab. 4. Test time is 0.451 s for 250 renders on a V100 (1 view) tab. 7.

Problem

• Implicit NeRF-style reconstructors are slow to render, up to 2 s for one 128 × 128 image §2.
• Voxel-based explicit methods are fast but scale poorly with resolution, and they assume absolute viewpoints §2.
• Triplane reconstructors reach large datasets only with hundreds of GPUs for several days §2.
• Gaussian Splatting is fast and high quality, but it needs many images per scene and had not been used in a learned, single-image setting §2.
• The method targets object-centric reconstruction with relative camera poses only, not generalisation to unseen scenes §2 app. B.

Method

Background. A radiance field assigns an opacity and a colour to each 3D point and viewing direction. Gaussian Splatting represents it as a set of coloured 3D Gaussians and renders by projecting and blending them §3.1 eq. 1 eq. 2. The renderer is a fast approximation of the volume-rendering integral §3.1.

Core idea. A network S maps an image I to a Gaussian mixture, the inverse of the renderer. The network predicts one Gaussian per input pixel; its output is an image-shaped tensor called the Splatter Image §3.2. Each pixel holds the opacity, offset, depth, scale, rotation and colour of its Gaussian §3.6.

Positions. A pixel at ray direction (u1, u2, 1) receives a depth d. The Gaussian centre is that ray point plus a predicted 3D offset, so the network can move a Gaussian off its ray to place it on hidden parts of the object §3.2 fig. 2 eq. 3.

Training. Pick a source image I and a target view J with pose change π. Predict the Splatter Image from I, render it at π, and minimise the squared error to J §3.3 eq. 4. Because a full image renders quickly, image-level losses such as LPIPS can be added §3.3. Regularisers keep Gaussians from becoming huge or vanishingly small; details are in the supplement §3.3 app. C.

Several views. Run the same network on each input view, bring each view's Gaussians into a common frame using the relative pose, and take the union of the sets §3.4. Each view is conditioned on its relative camera pose, and cross-view attention lets views exchange information at the lowest U-Net resolution §3.6.

View-dependent colour. Colours are either constant (Lambertian) or degree-1 spherical harmonics. When a view is rotated, the degree-1 coefficients are transformed by the rotation, conjugated by a fixed 3 × 3 permutation matrix, so that the colour seen from the new view stays consistent §3.5.

Network. The predictor is an EDM-style SongUNet §3.6 app. C whose last layer is a 1 × 1 convolution with 12 + k_c output channels, where k_c is 3 for Lambertian colour and 12 for degree-1 spherical harmonics §3.6. The channels are mapped to Gaussian parameters by these activations §3.6:

\sigma = \operatorname{sigmoid}(\hat\sigma) \qquad [\S 3.6]

d = (z_{\text{far}} - z_{\text{near}})\,\operatorname{sigmoid}(\hat d) + z_{\text{near}} \qquad [\S 3.6]

\Sigma = R(\mathbf q)\,\operatorname{diag}(\exp \hat{\mathbf s})^{2}\,R(\mathbf q)^{\top},\quad \mathbf q = \hat{\mathbf q}/\lVert\hat{\mathbf q}\rVert \qquad [\S 3.6]

Per-pixel Gaussian centre, from the rendering pose and the predicted offset:

\boldsymbol\mu = \begin{bmatrix} u_1 d + \Delta_x \\ u_2 d + \Delta_y \\ d + \Delta_z \end{bmatrix} \qquad [\text{eq. 3}]

Pseudocode for one view. Names follow the paper; unet is the SongUNet.

# Predict one Gaussian per pixel from one view  [§3.2] [§3.6]
# Input:  image (B, 3, H, W); view_to_world (B, 4, 4)
# Output: per-pixel Gaussian parameters, each (B, H * W, ...)
def predict_gaussians(image, view_to_world):
    # Run the image-to-image U-Net; 12 + k_c output channels  [§3.6]
    raw = unet(image)                                # (B, 12 + k_c, H, W)
    # Split the channels into the per-pixel parameters  [§3.6]
    sigma_hat, offset, depth_hat, scale_hat, quat_hat, colour_raw = split(raw)
    # Opacity lies in (0, 1) through a sigmoid  [§3.6]
    opacities = sigmoid(sigma_hat)
    # Depth lies between z_near and z_far through a sigmoid  [§3.6]
    depth = (z_far - z_near) * sigmoid(depth_hat) + z_near
    # Centre: ray point at depth d, plus the 3D offset  [§3.2] [eq. 3]
    means = ray_dirs * depth + offset
    # Scale is exp of the output; rotation is the normalised quaternion  [§3.6]
    scales = exp(scale_hat)
    rotations = quat_hat / norm(quat_hat)
    # Colour: Lambertian or degree-1 spherical harmonics  [§3.5] [§3.6]
    colours = colour_raw
    return means, scales, rotations, opacities, colours

Multi-view fusion. The pseudocode below follows the paper's union of warped mixtures §3.4. Covariances are rotated along with the centres §3.4 app. E.

# Fuse the Gaussians predicted from N views  [§3.4] [§3.5]
# Input:  per_view, a list of Gaussian sets; relative_pose, one per view
def fuse_views(per_view, relative_pose):
    fused = []
    for view_gaussians, pose in zip(per_view, relative_pose):
        # Move centres into the reference camera's frame  [§3.4]
        means = pose.rotation @ view_gaussians.means + pose.translation
        # Rotate each covariance with the same rotation  [§3.4] [app. E]
        covariances = pose.rotation @ view_gaussians.cov @ pose.rotation.T
        # Rotate the degree-1 spherical-harmonic coefficients  [§3.5]
        colours = rotate_sh(view_gaussians.colours, pose.rotation)
        fused.append(make_set(means, covariances, colours))
    # The union of the warped sets is the output mixture  [§3.4]
    return union(fused)

Training step. The paper's loss uses the conditioning view and the other sampled views app. C. The stage-2 blend follows app. C.

# One training step on one object  [§3.3] [eq. 4] [app. C]
# Input:  views, a list of (image, camera); views[0] is the input
# Output: scalar loss
def training_step(views, alpha):
    # Predict Gaussians from the input view only  [§3.3]
    gaussians = predict_gaussians(views[0].image, views[0].camera)
    # Render every target view, the input view included  [app. C]
    rendered = []
    for target in views:
        rendered.append(render(gaussians, target.camera))  # [§3.1]
    # Pixel-wise squared error, averaged over targets  [eq. 4]
    loss_l2 = mean_squared_error(rendered, [v.image for v in views])
    # Stage 1 uses alpha = 0; stage 2 blends in LPIPS  [app. C]
    loss_lpips = lpips(rendered, [v.image for v in views])
    return (1 - alpha) * loss_l2 + alpha * loss_lpips

Key equation, training loss (as printed; the paper writes the squared norm):

\mathcal{L}(\mathcal{S}) = \frac{1}{|\mathcal{D}|} \sum_{(I,J,\pi)\in\mathcal{D}} \bigl\lVert J - \mathcal{R}(\mathcal{S}(I),\pi) \bigr\rVert^{2} \qquad [\text{eq. 4}]

Hyperparameters

Values from the paper unless the "Where" column names code. "Not stated" means neither the paper nor the code gives a value; code-only values are marked.

Results

Author-reported. Setup: each benchmark's own protocol; baselines are the numbers the paper cites or re-runs §4 app. B. Higher PSNR and SSIM are better; lower LPIPS is better.

Single-view, ShapeNet-SRN, 128 × 128, relative pose for Ours tab. 1:

RC = method works with relative camera poses only tab. 1.

Multi-class ShapeNet, 64 × 64, NMR protocol tab. 2:

CO3D, hydrants and teddybears, single view, PixelNeRF trained by the authors for 400,000 iterations tab. 3 §4.1.1:

Google Scanned Objects, single view, Objaverse-trained model, Free3D renders tab. 4 §4.1.1:

Two views, ShapeNet-SRN cars, relative pose for Ours and PixelNeRF tab. 5:

Ablation, single view, shorter schedule tab. 6 app. C. The table caption does not name the dataset, so it is not stated.

Multi-view ablation (two views, cars) is in app. A: full model 24.11 PSNR, 0.92 SSIM, 0.087 LPIPS; removing cross-view attention, camera embedding or warping lowers PSNR to 23.68, 23.91 and 23.84 respectively.

Speed, ShapeNet-SRN, V100, 250 renders at 128² tab. 7 §4.2. E = encoding; R = one rendering; Forward = E + 4 R (training cost); Test = E + 250 R. Values in parentheses are self-reported by the NeRFDiff and FE-NVS authors, not measured by the paper §4.2. RP = relative pose.

Compute to train, as the paper reports it app. D: VisionNeRF 16 A100 × 5 days; NeRFDiff (estimated) 16 A100 × 3 days; ViewDiff 2 A40 × 3 days; PixelNeRF 1 TiRTX × 6 days; Splatter Image small-scale 1 A6000 × 7 days; LRM / OpenLRM (estimated) 128 A100 × 3 days; Splatter Image Objaverse 2 A6000 × 3.5 days.

Paper vs code

Code: official repository szymanowiczs/splatter-image at commit 78a6ad098e0cdc40c59c8ec98ca4fa439870fabd (cloned 2026-10-18). The owner is an author's account, and the paper's project page links it. The repository is BSD-3-Clause, but several files are adapted from non-commercial code (see the licensing bullet below).

• Exponential moving average. The paper does not mention EMA. The code keeps an EMA of the weights (decay 0.9999, update every 10 steps after step 100) default_config.yaml. Validation and the saved model_latest.pth use the EMA weights train_network.py train_network.py. The released numbers therefore depend on EMA.
• Loss normalisation. Eq. 4 writes a squared norm averaged over the dataset. The training script instead averages the squared error over every element of the rendered batch (pixels and channels) train_network.py loss_utils.py. That changes the loss scale, which Adam mostly absorbs.
• Adam epsilon. The code uses ε = 1 × 10⁻¹⁵ train_network.py. The paper gives no epsilon.
• Stage 2 restarts the optimiser. The checkpoint saves the optimiser state train_network.py, but the resume path loads only the model weights and the iteration counter train_network.py. The learning rate comes from the config file. The LPIPS stage therefore starts Adam from scratch.
• Multi-class ShapeNet schedule. The paper gives 800,000 L2 iterations then 150,000 with LPIPS app. C. The repository's multi-class base config sets 1,240,001 total iterations with batch 32 nmr.yaml, and the LPIPS config sets 150,001 lpips_nmr.yaml. The repo does not contain a config that matches the 800,000-iteration L2 stage in the paper.
• Objaverse LPIPS weight. The paper gives α = 0.338 app. C; the code uses 0.34 lpips_objaverse.yaml. Rounding, probably.
• CO3D regulariser thresholds. The paper's small-Gaussian threshold is −5 on the un-activated scale ŝ app. C. The code compares the activated scale with 1 × 10⁻⁵ train_network.py, which is about ŝ < −11.5. The code also multiplies both regularisers by 0.1 train_network.py. The paper gives neither the factor nor the mismatch.
• CO3D target views. The paper samples the input view plus three unseen views app. C. The code samples four frames at random, uses the first as input, and renders all four as targets, so the input view is one of the targets srn.py train_network.py. The count matches; the sampling is slightly different.
• Reference frame. The paper warps each view's Gaussians into the reference camera with (R, T) §3.4. The code converts each view's Gaussians into world coordinates with its view-to-world matrix gaussian_predictor.py, and the world frame is the first camera's frame after relative-pose normalisation shared_dataset.py. Rotations are composed with the view's quaternion gaussian_predictor.py gaussian_predictor.py. Same intent, different bookkeeping.
• Ray directions and focal length. Eq. 3 uses (u1, u2, 1) without saying whether u is normalised by focal length. For fixed-FOV datasets the code divides the x and y ray components by the focal length gaussian_predictor.py. For CO3D it uses per-sample focal lengths gaussian_predictor.py. The y axis is also inverted default_config.yaml gaussian_predictor.py.
• Camera conditioning. The paper says a sinusoidal embedding of "order 9" gives 60 dimensions §3.6. The code uses 10 channels per number, and six numbers, which is also 60 pose_pos.yaml. The code's (translation, third-axis) pair is built from matrix rows gaussian_predictor.py.
• SH transform. The paper gives Π for the degree-1 transform §3.5. The code uses a fixed 3 × 3 matrix in its own basis gaussian_predictor.py and tests it against random directions test_sh_transform.py. Use the code's matrix together with its SH basis; do not mix in Π.
• Depth for CO3D. The per-image origin distance is not fed to the U-Net. It is added to the depth as a constant gaussian_predictor.py gaussian_predictor.py. The paper says the centre of prediction is set to the object's centre app. B.
• Covariance. The paper defines the covariance from exp(ŝ) and a quaternion §3.6. The code passes exp(ŝ) and the normalised quaternion to the rasteriser, which builds the covariance itself gaussian_predictor.py [code: gaussian_renderer/init.py#L64-L69].
• Isotropic option. The code has an isotropic flag that repeats the first scale channel gaussian_predictor.py; the default is off default_config.yaml.
• Pruning. The paper says Gaussians with σ = 0 can be culled §3.2. I found no culling step in the training or evaluation paths read; the renderer simply receives them eval.py.
• Metrics. PSNR is computed per image, averaged over each object's renders, then averaged over objects eval.py eval.py. Images whose ground truth is all zero are skipped eval.py. The paper does not describe the averaging.
• Licensing of the code. scene/gaussian_predictor.py lines 1–428 are adapted from the EDM codebase and carry a CC BY-NC-SA 4.0 notice; utils/loss_utils.py and the rasteriser wrapper are adapted from an Inria codebase with non-commercial terms. Their behaviour is described here, but no code is reproduced. Implement the U-Net from the paper's description.

Pitfalls

• Equation typos. Eq. 1's denominator should sum over j, not reuse σ_i eq. 1. The offset vector is written (Δx, Δy, Δy), but Δz is meant §3.2.
• Opacity and scale initialisation matter. The initial opacity and scale biases set how much each Gaussian covers at the start default_config.yaml. The paper does not state them. The code uses different values for cars and chairs cars.yaml.
• EMA is needed to match the reported numbers, as explained under Paper vs code.
• FiLM is additive in the code. The paper says FiLM §3.6. The EDM-derived block with adaptive_scale = False adds a per-channel bias before a second norm and SiLU gaussian_predictor.py gaussian_predictor.py. A classic scale-and-shift FiLM will not reproduce the code. Keep the paper's description and check the effect on a small run.
• Cross-view attention reshapes features. Views are stacked along the height axis so that attention spans all views at the lowest resolution gaussian_predictor.py. The attention itself lives in the EDM-derived block, so implement it from the paper's description.
• Camera embedding detail. The paper's "order 9" does not map onto the code's helper, which uses 10 channels per number (5 frequencies with sine and cosine). Check the dimension (60) and the frequency schedule against the code.
• Two-view training is not reproducible from the repo. The repository has no two-view config; data.input_images defaults to 1 default_config.yaml. The two-view training settings are not stated in the paper.
• Speed claims need care.
  • The abstract's 38 FPS is not defined in the text. It matches 1 / 0.026 s, the encoding time E in tab. 7, but the paper does not say so.
  • The text says training is 246× faster than implicit methods §4.2. The table gives 7.432 / 0.033 ≈ 225× against PixelNeRF, and 9.733 / 0.033 ≈ 295× against VisionNeRF tab. 7.
  • The CO3D text says 1,000× faster §4.1.1; the Figure 4 caption says 100× fig. 4.
  • NeRFDiff and FE-NVS timings are the original authors' figures, on different hardware §4.2.
• Renderer speed. The 588 FPS rendering figure is in the abstract only.
• Memory. Memory is per GPU, and the Objaverse model needs about 26 GB per GPU §3.3. Batch size is limited by GPU memory app. C.
• Background. The paper says backgrounds are removed in CO3D preprocessing app. B but does not give a background colour for rendering. The code renders white for SRN, multi-class ShapeNet and Objaverse, and black for CO3D cars.yaml hydrants.yaml.
• Ablation dataset. The ablation caption does not name the dataset, and the ablation schedule is shorter than the main models app. C.

Lineage

• Builds on: 3D Gaussian Splatting, Kerbl et al. (SIGGRAPH 2023), for the representation and renderer §3.1 app. C; not a card in this wiki, and the paper does not give an arXiv link for it.
• Uses: the SongUNet architecture, cited as Song et al. 2020 in the paper's bibliography §3.6 app. C; the implementation is the EDM codebase of Karras et al. app. C. Also uses FiLM §3.6 and LPIPS §3.3.
• Compared against: ShapeNet-SRN and multi-class ShapeNet baselines (SRN, CodeNeRF, FE-NVS, ViewsetDiff, PixelNeRF, VisionNeRF, NeRFDiff) §4 tab. 1 tab. 2; PixelNeRF on CO3D tab. 3; OpenLRM (open source of LRM) on Google Scanned Objects tab. 4 §4.1.
• Superseded by: none known from content already in this wiki.