SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion

In plain words

Give SV3D a single photo of an object and it draws that object from a ring of camera positions, as if a camera had circled it. The same model can also be told where the camera should sit, so you can ask for views from a chosen height and angle. The authors start from a video generator, because video models already keep consecutive frames consistent, and they reuse that habit to keep the views of one object consistent with each other.

To turn those views into a 3D model, the authors first fit a rough shape, then refine it into a textured mesh. They separate the lighting from the surface colour, so the texture does not keep the shadows of the photo. For parts of the object that no generated view shows, they use a score-based loss from a diffusion model to fill in the gaps. The authors report that their views and meshes beat earlier single-image methods on the datasets they tested, and that people preferred their videos in a small user study.

TL;DR

• SV3D adapts Stable Video Diffusion (an image-to-video model) to generate 21 orbital views of an object from one 576x576 image §3.
• Two variants take a camera trajectory (SV3D-c, SV3D-p); SV3D-u takes only the image and makes a static orbit §3.
• The 3D step fits an Instant-NGP NeRF, then a DMTet mesh, with a disentangled illumination model and a visibility-masked SDS loss §4.
• All numbers are author-reported. The paper's NVS tables use GSO and OmniObject3D; 3D metrics use 50 GSO objects §3, §4.
• The paper's license is CC BY 4.0. The released code is MIT, but the released SV3D weights are under a non-commercial license (see Pitfalls).

Problem

• Single-image 3D generation from 2D diffusion models needs consistent views of the object from many angles. Methods that generate multiple views jointly are limited to specific views, or pay a high compute and data cost §1 §2, Multi-view Consistency.
• Image-based novel-view models such as EscherNet and Free3D generate 256x256 images §2. Earlier video-based methods (SVD-MV, IM-3D) produce at most 360 degrees at one elevation and cannot render arbitrary views §2.
• Setting: one image of an object, the object centred at the world origin, and the camera always looking at the centre. A viewpoint is therefore two numbers, elevation and azimuth §3, Problem Setting.
• Limits of the design: only 2 degrees of freedom; view inconsistency on mirror-like surfaces; and shading that is Lambertian, so reflective surfaces are not modelled §4, Limitations.

Method

Novel multi-view synthesis (SV3D)

The model generates an orbital video J of K = 21 frames, conditioned on one image I and a trajectory of (elevation, azimuth) pairs. It is trained to model p(J | I, trajectory) with a video diffusion model §3, Problem Setting.

The architecture starts from Stable Video Diffusion's UNet and makes four changes §3, SV3D Architecture:

1. The fps-id and motion-bucket-id vector conditions are removed §3 app. C.
2. The conditioning image is embedded by SVD's VAE encoder and concatenated to the noisy latent input at each step §3.
3. The CLIP embedding of the conditioning image is supplied as key and value to the cross-attention of every transformer block §3.
4. The camera trajectory is embedded: elevation, azimuth and the noise timestep each get a sinusoidal embedding. The camera embeddings are concatenated, passed through a linear layer and added to the timestep embedding. That sum is fed to every residual block, where it is added to the block output after another linear map §3.

Three models are trained §3, Models:

• SV3D-u: image only, no camera input, static orbit. The elevation is not given; the paper says the model infers it from the image.
• SV3D-c: image plus camera trajectory, trained on dynamic orbits.
• SV3D-p: first fine-tuned without camera input on static orbits, then fine-tuned on dynamic orbits with camera input. This follows the SVD practice of increasing task difficulty during training.

Orbit types §3, Static v.s. Dynamic Orbits:

• Static orbit: azimuths evenly spaced around the object, all at the elevation of the conditioning image.
• Dynamic orbit: azimuths are evenly spaced, then perturbed with small random noise. Elevation varies per view as a random weighted sum of sinusoids. The orbit loops back to the azimuth and elevation of the conditioning image, so it closes.

Triangular classifier-free guidance

SVD raises the guidance scale linearly from 1 to 4 across the frames. The authors observe that this over-sharpens the last frames of an orbit, which loops back to the front view. At inference they use a triangle wave instead: the weight climbs in a straight line from its minimum at the front view to its maximum at the back view, then descends symmetrically to the minimum again, so the two ends of the orbit match §3, Triangular CFG Scaling; fig. with linear vs triangle CFG in §3.

# Triangle CFG weight for each of the 21 frames  [§3]
# Output: scales (21,); 1.0 at frame 0, 2.5 at the middle frame, 1.0 at frame 20
# Frame positions run from 0 (front view) to 1 (back to front)  [§3]
def triangle_cfg_scales(num_frames=21, min_scale=1.0, max_scale=2.5):
    # Evenly spaced positions along the orbit  [§3]
    positions = linspace(0, 1, num_frames)              # (K,)
    # Triangle wave with period 1: 0 at the ends, 1 in the middle  [§3]
    wave = 2 * abs(positions - floor(positions + 0.5))  # (K,)
    # Map the wave onto the guidance range  [§3]
    return min_scale + (max_scale - min_scale) * wave   # (K,)

Sampling an orbit

The paper gives the inference steps and their count app. D. The sampler below follows the paper's description and the released code's structure, so the two are marked separately in the comments.

# Sample one 21-frame orbit  [§3] [app. D]
# Input:  image (3, 576, 576); camera (21, 2) of (elevation, azimuth), or None
# Output: frames (21, 3, 576, 576)

def sample_orbit(model, image, camera, num_steps=50):
    # Encode the conditioning image to a latent  [§3]
    image_latent = vae_encode(image)                    # (4, 72, 72)
    # Start all 21 frames from Gaussian noise  [§3]
    latents = randn(21, 4, 72, 72)                      # (K, C, h, w)
    # Noise levels for the sampling steps; EDM framework  [app. C] [app. D]
    sigmas = noise_schedule(num_steps)
    for step in range(num_steps):
        # Prediction with the image condition  [§3]
        pred_cond = model(latents, sigmas[step], image_latent, camera)
        # Prediction with the image condition dropped (not specified in paper)
        # [code: scripts/sampling/simple_video_sample.py#L224-L230]
        pred_free = model(latents, sigmas[step], None, camera)
        # Per-frame guidance weights from the triangle schedule  [§3]
        scales = triangle_cfg_scales(21)                # (K,)
        guided = pred_free + scales * (pred_cond - pred_free)
        # Deterministic Euler step to the next noise level  [app. D]
        latents = euler_step(latents, guided, sigmas[step], sigmas[step + 1])
    # Decode the latents back to frames  [§3]
    frames = vae_decode(latents)                        # (K, 3, 576, 576)
    # Released code overwrites the last frame with the input image  [code: L255]
    frames[-1] = image
    return frames

3D generation

The 3D pipeline takes the SV3D views and fits a mesh. The overview is in §4, 3D Optimization Pipeline figure. Its steps are §4:

1. Coarse stage. An Instant-NGP NeRF is trained to reproduce the SV3D-generated images (no SDS loss) at lower resolution.
2. Mesh extraction. A mesh is extracted from the NeRF with marching cubes.
3. Fine stage. A DMTet representation is fitted to the mesh at full resolution, using the photometric losses and, in its last stage, the masked SDS loss.
4. UV unwrapping. xatlas creates the UV map and the mesh is exported.

Reference orbit. The training orbit is one of the SV3D orbits (the reference orbit). A static orbit misses the top and bottom of the object. Therefore SV3D-c and SV3D-p are rendered on a dynamic orbit whose elevation follows a sine function §4, Training Orbits.

Disentangled illumination. The paper fits 24 spherical Gaussians as a white-light, Lambertian lighting model §4, Disentangled Illumination Model app. E. A loss pushes the value channel of the input image toward the rendered illumination §4. The lighting parameters are learned through the differentiable renderer from the SV3D images §4 app. E.

Photometric losses. The rendered images are compared with the SV3D images with a pixel MSE, a mask loss and an LPIPS loss §4 app. F. Geometry regularisers are a smooth depth loss (after RegNeRF), a bilateral normal-smoothness loss and a monocular normal loss from Omnidata §4 app. F. The albedo smoothness term uses a random 3D offset with scale 0.01 app. F.

Masked SDS. Plain SDS with SV3D as the diffusion guidance causes oversaturation and blur, so it is restricted to views the reference orbit does not see. A random orbit is rendered; for each visible surface point the reference camera with the largest dot product between view direction and normal is found. A soft mask, based on a smoothstep over [0, 0.5], is applied to the SDS loss §4, Masked SDS Loss.

# Soft visibility mask for the masked SDS loss  [§4] [app. F]
# Input:  points (N, 3); unit normals (N, 3); reference_positions (21, 3)
# Output: mask (N,); 0 where a reference view sees the point,
#         1 where it does not

def sds_mask(points, normals, reference_positions):
    # Vectors from each surface point to each reference camera  [§4]
    to_cameras = reference_positions - points[:, None]   # (N, 21, 3)
    views = to_cameras / norm(to_cameras, keepdim=True)  # (N, 21, 3)
    # Visibility score per reference camera: dot(view, normal)  [§4]
    scores = sum(views * normals[:, None], dim=-1)       # (N, 21)
    # Keep the reference camera with the highest score  [§4]
    best = max(scores, dim=1)                            # (N,)
    # Smoothstep over the range [0, 0.5], clipped to [0, 1]  [§4] [app. F]
    t = clip(best / 0.5, 0, 1)                           # (N,)
    return 1 - t * t * (3 - 2 * t)                       # (N,)

The masked SDS term is the per-pixel SDS loss multiplied by this mask §4. It is applied only in the last 200 iterations of the fine stage app. F.

Hyperparameters

Values marked "code" are taken from the released code, not the paper.

Results

Author-reported. Setup: each evaluation uses ground-truth orbit videos on the test objects. The last frame of the ground-truth video is the conditioning image [§3, Datasets]. Metrics are LPIPS, PSNR, SSIM, CLIP-S and MSE, compared against the ground-truth frames §3, Metrics.

Novel view synthesis, GSO static orbits §3, GSO static table

Novel view synthesis, GSO dynamic orbits §3, GSO dynamic table

Novel view synthesis, OmniObject3D static orbits §3, OmniObject3D static table

Novel view synthesis, OmniObject3D dynamic orbits §3, OmniObject3D dynamic table

User study §3: on 22 real-world images with 30 users, SV3D's static orbit was preferred over Zero123-XL, Stable Zero123, EscherNet and Free3D in 96%, 99%, 96% and 98% of comparisons, respectively.

2D comparison of 3D outputs, GSO §4, 2D table

3D metrics, GSO §4, 3D table

Ablation: models and training losses app. G.1

Ablation: training orbit for 3D app. G.2

Ablation: masked SDS variants app. G.3

Qualitative results (visual comparisons, per-frame LPIPS, real-world 3D meshes) are described in §3 §4 and app. H. Per-frame LPIPS is worst near the back view and best near the conditioning view §3, Quality Per Frame.

Paper vs code

Repository used. The paper links no code. The project page links only the Hugging Face weights. A GitHub search for "sv3d" found Stability-AI/generative-models, MIT-licensed (LICENSE-CODE), which contains SV3D inference configs and a sampling script. Its owner is Stability AI, the affiliation of all nine authors, so it is treated as official. Commit e8cd657656fa5d61688191730d0e03242bf4ed44.

Released weights are non-commercial. The SV3D checkpoints are covered by the Stability AI Non-Commercial Community License, not by the MIT code license LICENSE-SV3D. The paper's CC BY 4.0 license covers the paper only. Anyone implementing SV3D for commercial use needs their own rights to the weights.

No 3D code. The repository has no NeRF, DMTet, SDS, illumination or masked- SDS code. The 3D pipeline in §4 cannot be checked against code.

Training code is not released. The repository contains only inference code, so the training details in app. B and app. C cannot be checked against code.

Sampler name differs. The paper says 50 steps of a deterministic DDIM sampler app. D. The code uses an EDM Euler sampler with churn set to zero, and sets 50 steps by default sampling.py sv3d_p.yaml simple_video_sample.py.

Triangle CFG matches. The code uses a triangle guider with minimum 1.0, maximum 2.5 and period 1 over the frame positions, which peaks at the middle frame guiders.py sv3d_p.yaml. This matches §3.

Last frame replaced by the input. The released sampler overwrites the last generated frame with the conditioning image simple_video_sample.py. The paper does not mention this.

Conditioning noise. The code sets the conditioning-noise level to 1e-5 for both SV3D variants and adds it to the conditioning frame simple_video_sample.py simple_video_sample.py simple_video_sample.py. The paper does not state a value.

Elevation convention. The paper gives elevation and azimuth. The code feeds the polar angle, 90 degrees minus elevation, in radians simple_video_sample.py. The default elevation is 10 degrees simple_video_sample.py.

Frame order. Training orbits begin at azimuth 0 app. B. The released sampler uses 21 azimuths at 360/21-degree spacing, with azimuth 0 as the last frame, where the conditioning view is written simple_video_sample.py simple_video_sample.py.

Conditioning widths match. The 1280-channel conditioning of SV3D-p equals the 256 noise-level channels plus two 512-channel angle embeddings, which agrees with the concatenation described in §3 sv3d_p.yaml.

Preprocessing exists only in code. For inputs that are not RGBA, the script removes the background with rembg, crops around the object, resizes to 576x576 and composites on white simple_video_sample.py. The paper describes no test-time preprocessing for real photos.

Output post-processing exists only in code. Outputs are watermarked and passed through a detection filter simple_video_sample.py. The paper does not mention these steps.

Search log. GitHub search queries "sv3d" and the repository listing. The project page was opened and its links checked. No other official repository was found.

Pitfalls

• Released weights are non-commercial. See Paper vs code. Do not assume the CC BY 4.0 license or the MIT code license covers SV3D checkpoints.
• A claim in the paper is contradicted by its tables. The text of [§3, Ablative Analyses] says both SV3D-c and SV3D-p outperform SV3D-u on static orbits. The GSO static table gives SV3D-c PSNR 20.56 against 21.14 for SV3D-u, and the OmniObject3D static table gives 19.50 against 19.68. Only SV3D-p beats SV3D-u on PSNR.
• Sampler name is unresolved. The paper says DDIM, the code says Euler. Use the code's behaviour if you need to reproduce released outputs.
• Loss weights are missing. The compiled paper gives no numerical weights for the 3D losses app. F. The arXiv LaTeX source contains commented-out weights that the compiled paper does not show. This card does not use them.
• Illumination notation is ambiguous. The spherical Gaussian is defined with sharpness s, but its argument list names c. The order of the cosine-lobe parameters (2.133, 1.17) is not defined app. E.
• Masked SDS clipping is implied. The text says the smoothstep "clips" to the visibility range, and the formula has no explicit clamp to [0, 1]. The mask above clamps; the paper does not state it. The letter c is used both as the reference-camera index and as a constant in the spherical Gaussian §4 app. E.
• Mesh resolution is not stated. The coarse-stage resolution, the marching cubes resolution and the DMTet grid size are not given beyond "lower" and "full" resolution §4.
• Optimizer for SV3D training is not stated. The paper gives EDM and a simplified diffusion loss, but no learning rate, optimizer, warm-up, or noise range app. C.
• Distribution of random values is often not stated. The static-orbit elevation is "random" in [-5, 30] app. B. The SDS random-orbit sampling distribution is not stated §4.
• Frame 21 is the input in the released sampler. Check this if you compare frame-level numbers with the paper simple_video_sample.py.
• Noise range appears only in code. The released config sets sigma_max to 700 sv3d_p.yaml. The paper gives no noise range.
• Eval uses ground-truth conditioning. Metrics for NVS use the last ground-truth frame as input, so they assume a clean conditioning view §3, Datasets.
• Reflective surfaces and extra degrees of freedom are out of scope. The paper names mirror-like surfaces and extra camera degrees of freedom as limits §4, Limitations.
• Cost. Training took about 6 days on 32 A100 GPUs §3. A 21-frame batch fits 2 per GPU; 25 frames fit 1 app. C. Sampling takes about 40 seconds per orbit app. D. The 3D stage takes about 8 minutes without SDS and about 20 minutes with SDS §4.
• Training data. The training set is a filtered subset of Objaverse. The authors excluded objects tagged as restricted or PG-13 on Sketchfab, and they report that they found no content flagged as restricted app. A app. B.

Lineage

• Builds on: Stable Video Diffusion, arXiv:2311.15127, which SV3D fine-tunes §3. The EDM framework and the Objaverse dataset are also used app. C §3.
• Compared against: Zero123, Zero123-XL, SyncDreamer, Stable Zero123, Free3D, EscherNet §3; Point-E, Shap-E, One-2-3-45++, DreamGaussian, SyncDreamer, EscherNet, Free3D, Stable Zero123 §4.