Image-to-3D generation

Overview

The input is one photo; the output is a 3D model with a surface and a colour texture that matches the photo and holds up from other angles. The difficulty is that one photo shows only one side of an object. Per-view predictions that do not agree with each other distort the final shape 2311.07885 §1. Optimisation methods are slow, taking tens of minutes to hours per shape, while the feed-forward method that preceded this variant was fast but inconsistent across views 2311.07885 §1.

Most variants share one idea: generate several views of the subject together, so they agree, and then lift them to 3D in a second step. The second step differs by variant. One-2-3-45++ uses a learned 3D diffusion model over volumes 2311.07885 §3.2, PSHuman fits and remeshes a parametric body model to the generated views 2409.10141 §3.2, and SV3D fits a NeRF, then a DMTet mesh, to its generated orbit 2403.12008 §4. Make-It-3D and RealFusion take a different route. Neither generates views up front; each optimises a radiance field for one object. Make-It-3D uses score distillation at novel views and a loss at the reference view 2303.14184 §3.2. RealFusion uses score distillation on random views, prompted by a token learned from augmented copies of the input photo 2302.10663 §3.2. Human bodies add a further difficulty: the back and sides are hidden, and self-occluded poses are easily distorted 2409.10141 §1.

Variants

Multi-view diffusion, then sparse-volume 3D diffusion (One-2-3-45++)

Three modules run in sequence 2311.07885 §3.

• Consistent multi-view generation. A Stable Diffusion 2 model is fine-tuned to draw six views tiled into one 3×2 image, so the views are generated in one pass. The input enters through reference attention and a CLIP image embedding. Poses are fixed, so no input elevation is estimated 2311.07885 §3.1 2311.07885 fig. 3.
• 3D diffusion over volumes. Stage 1 denoises a 64³ occupancy volume. Each occupied voxel is split into 8, and stage 2 denoises a 128³ sparse volume that holds a signed distance and a colour per voxel. Both stages are conditioned on DINOv2 patch features, projected into each view with the known camera poses, then pooled over views 2311.07885 §3.2 2311.07885 fig. 4. Marching Cubes extracts the mesh 2311.07885 §3.2.
• Texture refinement. The geometry is fixed. A TensoRF colour field is fitted to the generated views with an L2 loss, then baked onto the mesh 2311.07885 §3.3.

Reported results, author-reported on the 1,030-shape GSO set with one frontal input: F-Score 93.6, CLIP similarity 81.0, user preference 87.6%, and about 60 s per object 2311.07885 tab. 1. Card: One-2-3-45++ (arXiv:2311.07885).

Multi-view diffusion with a body model, then SMPL-X remeshing (PSHuman)

This variant targets one full-body photo of a person. It also runs in two stages 2409.10141 §3.

• Body-face multi-view diffusion. Stable Diffusion 2.1-unclip is fine-tuned to generate six body views and one face view, as colour and normal maps 2409.10141 §3.1. The face tokens are blended into the front body view inside the self-attention blocks, so the face and body are generated together 2409.10141 §3.1.1 2409.10141 eq. 5. SMPL-X is estimated from the input and rendered from the six target viewpoints; a VAE encodes these renders and they are concatenated with the noise latent 2409.10141 §3.1.2. Training uses 2,385 THuman2.1 and 647 CustomHumans scans 2409.10141 §4.
• SMPL-X-initialised carving. SMPL-X translation, shape and pose are fitted so that its normal and silhouette renders match the generated views 2409.10141 eq. 7. Vertices and face connectivity are then optimised against a weighted normal and silhouette loss with a Laplacian smoothness term of weight 0.02 2409.10141 eq. 8. Poisson reconstruction fills small holes such as the chin 2409.10141 §3.2.
• Appearance. Per-vertex colours are fitted to the generated colour views 2409.10141 eq. 9; vertices that no view sees are filled by interpolation 2409.10141 §3.2.

Reported results, author-reported. With the ground-truth SMPL-X prior, the Chamfer distance on CAPE-NFP is 0.9688 against 1.5511 for the best baseline (ICON), and on THuman2.1 it is 0.4399 against 0.5754 (SIFU) 2409.10141 tab. 1. The introduction says the whole process takes about one minute 2409.10141 §1. The four reconstruction-module timings in the supplement add up to 54.1 s; the hardware is not stated 2409.10141 supp. §3. Card: PSHuman (arXiv:2409.10141).

Orbit video from a video model, then NeRF and mesh fitting (SV3D)

SV3D starts from Stable Video Diffusion, an image-to-video model, and generates a 21-frame orbit around the object from one 576×576 image 2403.12008 §3. SV3D-c and SV3D-p also take a camera trajectory of elevation and azimuth. SV3D-u takes only the image and makes a static orbit 2403.12008 §3. The guidance weight follows a triangle shape over the orbit: it rises as the camera moves toward the back, reaching 2.5 there, and falls again, so the two ends of the loop match 2403.12008 §3.

The 3D step has four parts 2403.12008 §4. A coarse Instant-NGP NeRF is trained on the generated frames without score distillation. A mesh is read from it with marching cubes. A DMTet representation is then fitted at full resolution, with a disentangled illumination model. In the last 200 iterations of that fine stage, a masked score-distillation loss is applied only to views the reference orbit does not see 2403.12008 app. F.

Reported results, author-reported. On GSO static orbits, SV3D-p reaches PSNR 21.26 against 18.34 for Stable Zero123, the best baseline in PSNR 2403.12008 §3. On 50 GSO objects, the Chamfer distance of its mesh is 0.024 against 0.039 for Stable Zero123 2403.12008 §4. Time: about 40 s per orbit 2403.12008 app. D, then about 8 min for the mesh without score distillation and about 20 min with it 2403.12008 §4. The released SV3D weights are non-commercial, which the paper's CC BY licence does not change; see the card's Paper vs code section. Card: SV3D (arXiv:2403.12008).

Per-object NeRF optimisation with a diffusion prior (Make-It-3D)

No views are generated up front. A NeRF is optimised for each object. Novel views get score distillation sampling from a Stable Diffusion prior with classifier-free guidance weight 10. At small timesteps a CLIP-based diffusion loss is used instead, with the switch at timestep 400 2303.14184 §3.2 2303.14184 app. C.1. The reference view gets a masked L1 loss, and a monocular depth estimate of the photo adds a scale-free correlation term 2303.14184 §3.2.

A refine stage exports the coarse NeRF at the reference view as a coloured point cloud, adds points for regions that the reference view does not show, and renders the cloud through a U-Net over multi-scale point descriptors. The geometry stays fixed 2303.14184 §3.3.

Reported results, author-reported. On DTU with one reference view, the enhanced model reaches LPIPS 0.0908 against 0.4075 for DreamFusion+. LPIPS is measured at the reference view, not at novel views 2303.14184 tab. 1. On the authors' own 400-image benchmark, CLIP similarity is 95.12% against 84.08% for DreamFusion+ 2303.14184 tab. 2. Training takes about 2 hours on one 32 GB V100 2303.14184 §4.1. Card: Make-It-3D (arXiv:2303.14184).

Per-object radiance field with a learned object token (RealFusion)

The input is one RGBA photo with the background removed. A single-image textual-inversion stage runs first. It learns one new token for the object from heavily augmented copies of the photo, with the diffusion model frozen 2302.10663 §3.2 2302.10663 app. A. Then an Instant-NGP radiance field is fitted. The input view is matched by an image term and a mask term at a fixed camera. Random prior views get score distillation through Stable Diffusion with guidance scale 100, and the prompt carries the learned token 2302.10663 §3.2 2302.10663 app. A. A smoothness term on the normals of the prior views, and a coarse-to-fine schedule over the feature levels, keep the surface stable 2302.10663 §3.2.

Reported results, author-reported. On CO3D, with seven categories and three images each, the mean F-score is 9.58 against 8.24 for Shelf-Supervised Mesh Prediction, and the mean CLIP similarity is 0.74 against 0.70 [arxiv:2302.10663 tab. 1]. The F-score is lower for skateboard and teddybear 2302.10663 tab. 1, and the sample is small 2302.10663 §4.2. On the lego novel-view test, the paper says the setting favours depth-supervised methods. RealFusion reaches PSNR 16.5 against 21.0 for SinNeRF 2302.10663 tab. 2 2302.10663 app. D. Reconstruction takes about 45 minutes on one V100, after the inversion stage 2302.10663 app. A. The authors report that the back sometimes copies the front, and that floating parts and the Janus problem remain 2302.10663 §4.3 2302.10663 app. F. Card: RealFusion (arXiv:2302.10663).

When to use which

All five variants have cards on this wiki. Baseline methods named in the papers have no cards; their numbers come from each paper's own tables.

Implementation sketch

No variant publishes a reference implementation of its full pipeline that this page can point to. One-2-3-45++ has only a README. PSHuman's code differs from its paper in texturing (see Contradictions). SV3D's code covers sampling, not the 3D stage, and its weights are non-commercial. Make-It-3D's code includes training but has no licence. RealFusion's code differs from its paper in several settings (see Contradictions). The sketch below follows the One-2-3-45++ description and is not a reference implementation.

# Image-to-3D with a joint six-view tile and two 3D diffusion stages  [arxiv:2311.07885 §3]
# Input:  one RGB image (H, W, 3)
# Output: textured mesh

def image_to_3d(image):
    # Draw six fixed-pose views as one 3x2 tile  [arxiv:2311.07885 §3.1]
    tile = multi_view_diffusion(image)
    views, poses = split_tile(tile)
    # Patch features for each view  [arxiv:2311.07885 §3.2]
    features = dino_patch_features(views)
    # Stage 1: coarse occupancy at 64^3  [arxiv:2311.07885 §3.2]
    occupancy = diffuse_occupancy(features, poses, size=64)
    # Stage 2: split occupied voxels into 8 and denoise SDF and colour  [arxiv:2311.07885 §3.2]
    sparse = subdivide(occupancy)
    sdf_and_colour = diffuse_sparse(features, poses, sparse, size=128)
    # Extract the surface  [arxiv:2311.07885 §3.2]
    mesh = marching_cubes(sdf_and_colour)
    # Fit a colour field to the views; geometry stays fixed  [arxiv:2311.07885 §3.3]
    return refine_texture(mesh, views, poses)

Where PSHuman differs 2409.10141 §3:

• The multi-view step is conditioned on SMPL-X renders as well as the input image, and its output is a set of colour and normal maps.
• The 3D step is not a diffusion model. The SMPL-X fit 2409.10141 eq. 7 is the starting mesh, and remeshing against the normal and silhouette maps 2409.10141 eq. 8 produces the final geometry.
• Texture is a per-vertex fit to the colour views 2409.10141 eq. 9, not a colour field.

Where SV3D differs 2403.12008 §3 2403.12008 §4:

• The multi-view step is one video-diffusion pass over 21 frames, conditioned on the image and, in SV3D-c and SV3D-p, on a camera trajectory. There is no tiling into one image and no volume diffusion.
• The 3D step fits a NeRF to the generated frames, then a DMTet mesh, with a masked score-distillation term at the end. It does not use a volume network.

Where Make-It-3D differs 2303.14184 §3:

• There is no multi-view generation step. The 3D field is optimised directly for each object, with score distillation at novel views and a masked L1 loss at the reference view.
• The second stage refines a coloured point cloud with a learned renderer 2303.14184 §3.3. It does not fit a mesh to generated views.

Where RealFusion differs 2302.10663 §3.2 2302.10663 app. A:

• There is no multi-view step. The input view supervises a fixed camera, and the diffusion model acts only through score distillation on random views.
• A separate inversion stage learns the object token before the field is fitted, so the prompt names this object rather than its category 2302.10663 §4.4.
• The output is a radiance field, read out with marching cubes 2302.10663 §4.2. It is not fitted to a parametric body or a DMTet mesh.

Settings the papers do not state, which an implementer has to choose:

• One-2-3-45++: learning rates, optimisers, batch sizes, and the LoRA rank 2311.07885 §3.1; diffusion sampler, step count and guidance scale 2311.07885 §3; the noise formula, the occupancy threshold and the 3D noise schedule 2311.07885 §3.2; TensoRF resolution and iteration count 2311.07885 §3.3; and the camera convention 2311.07885 §3.2.
• PSHuman: the optimiser for the SMPL-X fit, the Poisson depth, and the classifier-free guidance scale. The paper does not state them 2409.10141 §4; the released code sets some of them, which is why the paper alone is not enough to reproduce the run.
• SV3D: the loss weights of the 3D stage 2403.12008 app. F; the camera distance rule and the distribution of static-orbit elevations 2403.12008 app. B.
• Make-It-3D: the loss weights, which the paper does not give 2303.14184 §3.2; the learning-rate schedule 2303.14184 §4.1; and the CLIP model used for the CLIP-based loss.
• RealFusion: the Adam betas and batch size of the reconstruction stage, the normal blur sigma and finite-difference step, the matting model that makes the mask, and the timestep weight w(t) 2302.10663 app. A 2302.10663 §3.1 2302.10663 §3.2.

Contradictions

• One-2-3-45++: claim vs table. The text says the joint multi-view step beats prior methods on PSNR, LPIPS and mask IoU 2311.07885 §4.3. The same paper's table shows SyncDreamer ahead on LPIPS (0.095 vs 0.110) and mask IoU (0.894 vs 0.878), and the methods use different target elevations 2311.07885 tab. 5.
• SV3D: claim vs table. The text says both SV3D-c and SV3D-p outperform SV3D-u on static orbits 2403.12008 §3. The GSO static table gives SV3D-c PSNR 20.56 against 21.14 for SV3D-u, and the OmniObject3D table gives 19.50 against 19.68 2403.12008 §3. Only SV3D-p beats SV3D-u on PSNR.
• SV3D: sampler name. The paper says 50 steps of a deterministic DDIM sampler 2403.12008 app. D. The released code uses an EDM Euler sampler with churn set to zero sampling.py. Use the code's behaviour if you need to reproduce released outputs.
• PSHuman: paper vs its own code, texture. The paper describes per-vertex colour optimisation over 100 steps at learning rate 0.0005 2409.10141 §4 2409.10141 eq. 9. The released code projects the generated views onto the mesh with a confidence threshold, and a code comment says it departs from the paper reconstruct.py reconstruct.py. Implement one or the other, not both.
• PSHuman: paper vs its own code, optimisation steps. The paper's §4 gives 700 / 100 / 100 steps for alignment, geometry and texture, while §3.2 says remeshing takes 700 steps 2409.10141 §4 2409.10141 §3.2. The code runs 100 alignment steps and 700 remeshing steps reconstruct.py remesh.yaml.
• Cross-paper: One-2-3-45++ against SV3D on 3D quality. The One-2-3-45++ table ranks it first on F-Score over 1,030 GSO shapes, ahead of Zero123 XL at 91.6 2311.07885 tab. 1. SV3D's 3D table, on 50 GSO objects, gives One-2-3-45++ a Chamfer distance of 0.054 and 3D IoU of 0.406, behind Stable Zero123 at 0.039 and 0.550 2403.12008 §4. The metrics and object sets differ, so the two rankings are not reconciled here.
• Timings across the variants. One-2-3-45++ reports 60 s 2311.07885 tab. 1, PSHuman about one minute 2409.10141 §1, SV3D about 40 s per orbit 2403.12008 app. D, Make-It-3D about 2 hours of training 2303.14184 §4.1, and RealFusion about 45 minutes of reconstruction on one V100 2302.10663 app. A. The hardware is not stated for the first three, or differs, and the last two measure different things. The times cannot be compared.
• RealFusion: paper vs its own code. The paper places the coarse-to-fine switch halfway through training, about step 2500 of 5000 2302.10663 §3.2 2302.10663 app. A. The code lifts the masking only after step 3000 options.py. The paper's front band for prior cameras is ±30° of azimuth, while the code uses absolute azimuth 2302.10663 app. A provider_synthetic.py. The paper runs camera replacement every tenth step, while the code runs it on nine of every ten 2302.10663 app. A trainer.py.

Papers

• One-2-3-45++ (arXiv:2311.07885): joint six-view tile from a fine-tuned Stable Diffusion 2, two sparse-volume 3D diffusion stages and a TensoRF texture pass.
• PSHuman (arXiv:2409.10141): body and face multi-view diffusion conditioned on SMPL-X renders, then SMPL-X-initialised remeshing for full-body humans.
• SV3D (arXiv:2403.12008): Stable Video Diffusion fine-tuned to render a camera-steered orbit, then a NeRF and DMTet fit with masked score distillation.
• Make-It-3D (arXiv:2303.14184): per-object NeRF optimised with a diffusion prior, then point-cloud refinement with a learned renderer.
• RealFusion (arXiv:2302.10663): one RGBA photo, a radiance field fitted with score distillation on random views, prompted by a token learned from augmented copies of the photo.