---
kind: concept
slug: image-to-3d-generation
title: Image-to-3D generation
summary: >-
  Turning one photo into a 3D model, by generating views or by optimising a 3D field with a 2D diffusion prior. Five variants: One-2-3-45++ (objects, about 60 s, author-reported), PSHuman (full-body people, SMPL-X), SV3D (orbit video, non-commercial weights), Make-It-3D and RealFusion (per-object optimisation, tens of minutes to hours).
tags: [3d-generation, multi-view-diffusion, mesh, 3d-human-reconstruction, smpl-x, nerf, video-diffusion, textual-inversion]
papers: ["2311.07885", "2409.10141", "2403.12008", "2303.14184", "2302.10663"]
related: [diffusion-models, multi-view-reconstruction, novel-view-synthesis, latent-diffusion, differentiable-rendering, low-rank-adaptation, score-distillation-sampling, neural-radiance-field, classifier-free-guidance, video-generation]
status: draft
updated_at: 2026-11-01
---

## Overview

The input is one photo; the output is a 3D model with a surface and a colour
texture that matches the photo and holds up from other angles. The difficulty
is that one photo shows only one side of an object. Per-view predictions that
do not agree with each other distort the final shape [arxiv:2311.07885 §1].
Optimisation methods are slow, taking tens of minutes to hours per shape, while
the feed-forward method that preceded this variant was fast but inconsistent
across views [arxiv:2311.07885 §1].

Most variants share one idea: generate several views of the subject together,
so they agree, and then lift them to 3D in a second step. The second step
differs by variant. One-2-3-45++ uses a learned 3D diffusion model over volumes
[arxiv:2311.07885 §3.2], PSHuman fits and remeshes a parametric body model to
the generated views [arxiv:2409.10141 §3.2], and SV3D fits a NeRF, then a DMTet
mesh, to its generated orbit [arxiv:2403.12008 §4]. Make-It-3D and RealFusion
take a different route. Neither generates views up front; each optimises a
radiance field for one object. Make-It-3D uses score distillation at novel views
and a loss at the reference view [arxiv:2303.14184 §3.2]. RealFusion uses score
distillation on random views, prompted by a token learned from augmented copies
of the input photo [arxiv:2302.10663 §3.2]. Human bodies add a further
difficulty: the back and
sides are hidden, and self-occluded poses are easily distorted
[arxiv:2409.10141 §1].

## Variants

### Multi-view diffusion, then sparse-volume 3D diffusion (One-2-3-45++)

Three modules run in sequence [arxiv:2311.07885 §3].

- **Consistent multi-view generation.** A Stable Diffusion 2 model is
  fine-tuned to draw six views tiled into one 3×2 image, so the views are
  generated in one pass. The input enters through reference attention and a
  CLIP image embedding. Poses are fixed, so no input elevation is estimated
  [arxiv:2311.07885 §3.1] [arxiv:2311.07885 fig. 3].
- **3D diffusion over volumes.** Stage 1 denoises a 64³ occupancy volume.
  Each occupied voxel is split into 8, and stage 2 denoises a 128³ sparse
  volume that holds a signed distance and a colour per voxel. Both stages
  are conditioned on DINOv2 patch features, projected into each view with
  the known camera poses, then pooled over views [arxiv:2311.07885 §3.2]
  [arxiv:2311.07885 fig. 4]. Marching Cubes extracts the mesh
  [arxiv:2311.07885 §3.2].
- **Texture refinement.** The geometry is fixed. A TensoRF colour field is
  fitted to the generated views with an L2 loss, then baked onto the mesh
  [arxiv:2311.07885 §3.3].

Reported results, author-reported on the 1,030-shape GSO set with one frontal
input: F-Score 93.6, CLIP similarity 81.0, user preference 87.6%, and about 60 s
per object [arxiv:2311.07885 tab. 1]. Card: [One-2-3-45++ (arXiv:2311.07885)](/papers/2311.07885).

### Multi-view diffusion with a body model, then SMPL-X remeshing (PSHuman)

This variant targets one full-body photo of a person. It also runs in two
stages [arxiv:2409.10141 §3].

- **Body-face multi-view diffusion.** Stable Diffusion 2.1-unclip is
  fine-tuned to generate six body views and one face view, as colour and
  normal maps [arxiv:2409.10141 §3.1]. The face tokens are blended into the
  front body view inside the self-attention blocks, so the face and body are
  generated together [arxiv:2409.10141 §3.1.1] [arxiv:2409.10141 eq. 5]. SMPL-X
  is estimated from the input and rendered from the six target viewpoints; a
  VAE encodes these renders and they are concatenated with the noise latent
  [arxiv:2409.10141 §3.1.2]. Training uses 2,385 THuman2.1 and 647
  CustomHumans scans [arxiv:2409.10141 §4].
- **SMPL-X-initialised carving.** SMPL-X translation, shape and pose are
  fitted so that its normal and silhouette renders match the generated views
  [arxiv:2409.10141 eq. 7]. Vertices and face connectivity are then optimised
  against a weighted normal and silhouette loss with a Laplacian smoothness
  term of weight 0.02 [arxiv:2409.10141 eq. 8]. Poisson reconstruction fills
  small holes such as the chin [arxiv:2409.10141 §3.2].
- **Appearance.** Per-vertex colours are fitted to the generated colour views
  [arxiv:2409.10141 eq. 9]; vertices that no view sees are filled by
  interpolation [arxiv:2409.10141 §3.2].

Reported results, author-reported. With the ground-truth SMPL-X prior, the
Chamfer distance on CAPE-NFP is 0.9688 against 1.5511 for the best baseline
(ICON), and on THuman2.1 it is 0.4399 against 0.5754 (SIFU)
[arxiv:2409.10141 tab. 1]. The introduction says the whole process takes about
one minute [arxiv:2409.10141 §1]. The four reconstruction-module timings in the
supplement add up to 54.1 s; the hardware is not stated
[arxiv:2409.10141 supp. §3]. Card: [PSHuman (arXiv:2409.10141)](/papers/2409.10141).

### Orbit video from a video model, then NeRF and mesh fitting (SV3D)

SV3D starts from Stable Video Diffusion, an image-to-video model, and generates
a 21-frame orbit around the object from one 576×576 image [arxiv:2403.12008 §3].
SV3D-c and SV3D-p also take a camera trajectory of elevation and azimuth. SV3D-u
takes only the image and makes a static orbit [arxiv:2403.12008 §3]. The
guidance weight follows a triangle shape over the orbit: it rises as the camera
moves toward the back, reaching 2.5 there, and falls again, so the two ends of
the loop match [arxiv:2403.12008 §3].

The 3D step has four parts [arxiv:2403.12008 §4]. A coarse Instant-NGP NeRF is
trained on the generated frames without score distillation. A mesh is read
from it with marching cubes. A DMTet representation is then fitted at full
resolution, with a disentangled illumination model. In the last 200 iterations
of that fine stage, a masked score-distillation loss is applied only to views
the reference orbit does not see [arxiv:2403.12008 app. F].

Reported results, author-reported. On GSO static orbits, SV3D-p reaches PSNR
21.26 against 18.34 for Stable Zero123, the best baseline in PSNR
[arxiv:2403.12008 §3]. On 50 GSO objects, the Chamfer distance of its mesh is
0.024 against 0.039 for Stable Zero123 [arxiv:2403.12008 §4]. Time: about 40 s
per orbit [arxiv:2403.12008 app. D], then about 8 min for the mesh without
score distillation and about 20 min with it [arxiv:2403.12008 §4]. The
released SV3D weights are non-commercial, which the paper's CC BY licence does
not change; see the card's Paper vs code section. Card: [SV3D (arXiv:2403.12008)](/papers/2403.12008).

### Per-object NeRF optimisation with a diffusion prior (Make-It-3D)

No views are generated up front. A NeRF is optimised for each object. Novel
views get score distillation sampling from a Stable Diffusion prior with
classifier-free guidance weight 10. At small timesteps a CLIP-based diffusion
loss is used instead, with the switch at timestep 400 [arxiv:2303.14184 §3.2]
[arxiv:2303.14184 app. C.1]. The reference view gets a masked L1 loss, and a
monocular depth estimate of the photo adds a scale-free correlation term
[arxiv:2303.14184 §3.2].

A refine stage exports the coarse NeRF at the reference view as a coloured
point cloud, adds points for regions that the reference view does not show,
and renders the cloud through a U-Net over multi-scale point descriptors. The
geometry stays fixed [arxiv:2303.14184 §3.3].

Reported results, author-reported. On DTU with one reference view, the enhanced
model reaches LPIPS 0.0908 against 0.4075 for DreamFusion+. LPIPS is measured at
the reference view, not at novel views [arxiv:2303.14184 tab. 1]. On the
authors' own 400-image benchmark, CLIP similarity is 95.12% against 84.08% for
DreamFusion+ [arxiv:2303.14184 tab. 2]. Training takes about 2 hours on one
32 GB V100 [arxiv:2303.14184 §4.1]. Card: [Make-It-3D (arXiv:2303.14184)](/papers/2303.14184).

### Per-object radiance field with a learned object token (RealFusion)

The input is one RGBA photo with the background removed. A single-image
textual-inversion stage runs first. It learns one new token for the object from
heavily augmented copies of the photo, with the diffusion model frozen
[arxiv:2302.10663 §3.2] [arxiv:2302.10663 app. A]. Then an Instant-NGP radiance
field is fitted. The input view is matched by an image term and a mask term at
a fixed camera. Random prior views get score distillation through Stable
Diffusion with guidance scale 100, and the prompt carries the learned token
[arxiv:2302.10663 §3.2] [arxiv:2302.10663 app. A]. A smoothness term on the
normals of the prior views, and a coarse-to-fine schedule over the feature
levels, keep the surface stable [arxiv:2302.10663 §3.2].

Reported results, author-reported. On CO3D, with seven categories and three
images each, the mean F-score is 9.58 against 8.24 for Shelf-Supervised Mesh
Prediction, and the mean CLIP similarity is 0.74 against 0.70 [arxiv:2302.10663
tab. 1]. The F-score is lower for skateboard and teddybear
[arxiv:2302.10663 tab. 1], and the sample is small [arxiv:2302.10663 §4.2]. On
the lego novel-view test, the paper says the setting favours depth-supervised
methods. RealFusion reaches PSNR 16.5 against 21.0 for SinNeRF
[arxiv:2302.10663 tab. 2] [arxiv:2302.10663 app. D]. Reconstruction takes about
45 minutes on one V100, after the inversion stage [arxiv:2302.10663 app. A].
The authors report that the back sometimes copies the front, and that floating
parts and the Janus problem remain [arxiv:2302.10663 §4.3]
[arxiv:2302.10663 app. F]. Card: [RealFusion (arXiv:2302.10663)](/papers/2302.10663).

## When to use which

All five variants have cards on this wiki. Baseline methods named in the papers
have no cards; their numbers come from each paper's own tables.

| Situation | Use | Why |
|---|---|---|
| Input is a person, full body visible, and an SMPL-X fit is acceptable | PSHuman | Built for this case; reports lower Chamfer distance than the best baseline in its tables [arxiv:2409.10141 tab. 1]. The SMPL-X models have their own licence terms, which the card leaves unchecked |
| Input is an object, you need a mesh, and the repository is not a requirement | One-2-3-45++ | PSHuman's body prior does not apply. One-2-3-45++ reports F-Score 93.6 and CLIP similarity 81.0 on GSO [arxiv:2311.07885 tab. 1]. Its repository holds only a README |
| Input is an object, and you need views on a chosen camera path more than a mesh | SV3D-p | Takes a camera trajectory. Reports PSNR 21.26 on GSO static orbits against 18.34 for Stable Zero123 [arxiv:2403.12008 §3]. The released weights are non-commercial |
| Textured mesh from one photo, about a minute per object is acceptable | One-2-3-45++ or PSHuman | Both report about one minute, author-reported [arxiv:2311.07885 tab. 1] [arxiv:2409.10141 §1]. Hardware is not stated in either, so the times are not directly comparable |
| Speed matters more than texture match, for objects | Shap-E, a baseline in the One-2-3-45++ table (no card here) | Reported 27 s, F-Score 91.8, CLIP similarity 73.1 [arxiv:2311.07885 tab. 1] |
| Reference view must match closely, and hours per object are acceptable | Make-It-3D | Reports LPIPS 0.0908 at the reference view on DTU, against 0.4075 for DreamFusion+ [arxiv:2303.14184 tab. 1]. About 2 hours of optimisation per object on one V100 [arxiv:2303.14184 §4.1] |
| Input is one masked object photo, you need a 360° model, no multi-view model is available, and tens of minutes to hours per object is acceptable | RealFusion | Reports mean CO3D F-score 9.58 against 8.24 for Shelf-Supervised Mesh Prediction [arxiv:2302.10663 tab. 1]. Needs an RGBA input and a separate inversion stage before reconstruction [arxiv:2302.10663 app. A] |
| Novel-view PSNR matters more than shape, or depth is available | A depth-supervised NeRF such as SinNeRF (no card here) | RealFusion reports PSNR 16.5 against 21.0 on the lego test set, a setting the paper says favours depth supervision [arxiv:2302.10663 tab. 2] |
| You need code to run now | PSHuman | Official code is published under MIT [code: reconstruct.py#L387]. One-2-3-45++ has only a README, so its pipeline must be built from the paper. Make-It-3D's code has no LICENSE file |
| Commercial product that ships model weights | Check each licence first | SV3D's released weights are non-commercial, though its inference code is MIT. PSHuman's SMPL-X models have their own terms, not checked. Make-It-3D's code has no LICENSE file, so cite it by path and line only. RealFusion's code is Apache-2.0, but the Stable Diffusion v1.5 checkpoint it loads has a licence that is not checked here [code: nerf/options.py#L33] |
| Training budget is tight, objects | Check before choosing One-2-3-45++ or SV3D | One-2-3-45++ stage 1 used 16 GPUs for about 10 days, and each 3D stage used 8 A100s for about 10 days [arxiv:2311.07885 §3.1] [arxiv:2311.07885 §3.2]. SV3D used 32 A100s for about 6 days [arxiv:2403.12008 §3]. Make-It-3D trains nothing across objects, but takes about 2 hours per object. RealFusion also trains nothing across objects; its reconstruction takes about 45 minutes on one V100 after the inversion stage [arxiv:2302.10663 app. A] |
| Training budget is tight, humans | Check before choosing PSHuman | Training reported as about 3 days on 16 H800 GPUs [arxiv:2409.10141 §4] |
| Texture refinement is too costly or not needed, objects | Skip it only if colour fidelity matters less | Without texture refinement CLIP similarity falls from 81.0 to 73.4; F-Score is unchanged at 93.6 [arxiv:2311.07885 tab. 3] |
| Multi-view step must use a per-view model you already run, objects | Zero123 XL in place of the joint tile | Ablation with per-view Zero123 XL gives F-Score 92.9 at 14 s, against 93.6 at 20 s for the joint tile (both without texture) [arxiv:2311.07885 tab. 3] |
| Text-only input | Not covered | One-2-3-45++ reports text-to-3D numbers, but does not say how text becomes the input image for its pipeline [arxiv:2311.07885 §4.2]. Make-It-3D shows text-to-3D by first generating a reference image, in a figure only [arxiv:2303.14184 §5] |

## Implementation sketch

No variant publishes a reference implementation of its full pipeline that this
page can point to. One-2-3-45++ has only a README. PSHuman's code differs from
its paper in texturing (see Contradictions). SV3D's code covers sampling, not
the 3D stage, and its weights are non-commercial. Make-It-3D's code includes
training but has no licence. RealFusion's code differs from its paper in
several settings (see Contradictions). The sketch below follows the One-2-3-45++
description and is not a reference implementation.

```pseudocode
# Image-to-3D with a joint six-view tile and two 3D diffusion stages  [arxiv:2311.07885 §3]
# Input:  one RGB image (H, W, 3)
# Output: textured mesh

def image_to_3d(image):
    # Draw six fixed-pose views as one 3x2 tile  [arxiv:2311.07885 §3.1]
    tile = multi_view_diffusion(image)
    views, poses = split_tile(tile)
    # Patch features for each view  [arxiv:2311.07885 §3.2]
    features = dino_patch_features(views)
    # Stage 1: coarse occupancy at 64^3  [arxiv:2311.07885 §3.2]
    occupancy = diffuse_occupancy(features, poses, size=64)
    # Stage 2: split occupied voxels into 8 and denoise SDF and colour  [arxiv:2311.07885 §3.2]
    sparse = subdivide(occupancy)
    sdf_and_colour = diffuse_sparse(features, poses, sparse, size=128)
    # Extract the surface  [arxiv:2311.07885 §3.2]
    mesh = marching_cubes(sdf_and_colour)
    # Fit a colour field to the views; geometry stays fixed  [arxiv:2311.07885 §3.3]
    return refine_texture(mesh, views, poses)
```

Where PSHuman differs [arxiv:2409.10141 §3]:

- The multi-view step is conditioned on SMPL-X renders as well as the input
  image, and its output is a set of colour and normal maps.
- The 3D step is not a diffusion model. The SMPL-X fit [arxiv:2409.10141 eq. 7]
  is the starting mesh, and remeshing against the normal and silhouette maps
  [arxiv:2409.10141 eq. 8] produces the final geometry.
- Texture is a per-vertex fit to the colour views [arxiv:2409.10141 eq. 9], not
  a colour field.

Where SV3D differs [arxiv:2403.12008 §3] [arxiv:2403.12008 §4]:

- The multi-view step is one video-diffusion pass over 21 frames, conditioned on
  the image and, in SV3D-c and SV3D-p, on a camera trajectory. There is no
  tiling into one image and no volume diffusion.
- The 3D step fits a NeRF to the generated frames, then a DMTet mesh, with a
  masked score-distillation term at the end. It does not use a volume network.

Where Make-It-3D differs [arxiv:2303.14184 §3]:

- There is no multi-view generation step. The 3D field is optimised directly for
  each object, with score distillation at novel views and a masked L1 loss at
  the reference view.
- The second stage refines a coloured point cloud with a learned renderer
  [arxiv:2303.14184 §3.3]. It does not fit a mesh to generated views.

Where RealFusion differs [arxiv:2302.10663 §3.2] [arxiv:2302.10663 app. A]:

- There is no multi-view step. The input view supervises a fixed camera, and
  the diffusion model acts only through score distillation on random views.
- A separate inversion stage learns the object token before the field is
  fitted, so the prompt names this object rather than its category
  [arxiv:2302.10663 §4.4].
- The output is a radiance field, read out with marching cubes
  [arxiv:2302.10663 §4.2]. It is not fitted to a parametric body or a DMTet
  mesh.

Settings the papers do not state, which an implementer has to choose:

- One-2-3-45++: learning rates, optimisers, batch sizes, and the LoRA rank
  [arxiv:2311.07885 §3.1]; diffusion sampler, step count and guidance scale
  [arxiv:2311.07885 §3]; the noise formula, the occupancy threshold and the 3D
  noise schedule [arxiv:2311.07885 §3.2]; TensoRF resolution and iteration
  count [arxiv:2311.07885 §3.3]; and the camera convention
  [arxiv:2311.07885 §3.2].
- PSHuman: the optimiser for the SMPL-X fit, the Poisson depth, and the
  classifier-free guidance scale. The paper does not state them
  [arxiv:2409.10141 §4]; the released code sets some of them, which is why the
  paper alone is not enough to reproduce the run.
- SV3D: the loss weights of the 3D stage [arxiv:2403.12008 app. F]; the camera
  distance rule and the distribution of static-orbit elevations
  [arxiv:2403.12008 app. B].
- Make-It-3D: the loss weights, which the paper does not give
  [arxiv:2303.14184 §3.2]; the learning-rate schedule [arxiv:2303.14184 §4.1];
  and the CLIP model used for the CLIP-based loss.
- RealFusion: the Adam betas and batch size of the reconstruction stage, the
  normal blur sigma and finite-difference step, the matting model that makes
  the mask, and the timestep weight w(t) [arxiv:2302.10663 app. A]
  [arxiv:2302.10663 §3.1] [arxiv:2302.10663 §3.2].

## Contradictions

- **One-2-3-45++: claim vs table.** The text says the joint multi-view step
  beats prior methods on PSNR, LPIPS and mask IoU [arxiv:2311.07885 §4.3]. The
  same paper's table shows SyncDreamer ahead on LPIPS (0.095 vs 0.110) and mask
  IoU (0.894 vs 0.878), and the methods use different target elevations
  [arxiv:2311.07885 tab. 5].
- **SV3D: claim vs table.** The text says both SV3D-c and SV3D-p outperform
  SV3D-u on static orbits [arxiv:2403.12008 §3]. The GSO static table gives
  SV3D-c PSNR 20.56 against 21.14 for SV3D-u, and the OmniObject3D table gives
  19.50 against 19.68 [arxiv:2403.12008 §3]. Only SV3D-p beats SV3D-u on PSNR.
- **SV3D: sampler name.** The paper says 50 steps of a deterministic DDIM
  sampler [arxiv:2403.12008 app. D]. The released code uses an EDM Euler
  sampler with churn set to zero [code: sgm/modules/diffusionmodules/sampling.py#L82-L91].
  Use the code's behaviour if you need to reproduce released outputs.
- **PSHuman: paper vs its own code, texture.** The paper describes per-vertex
  colour optimisation over 100 steps at learning rate 0.0005
  [arxiv:2409.10141 §4] [arxiv:2409.10141 eq. 9]. The released code projects the
  generated views onto the mesh with a confidence threshold, and a code comment
  says it departs from the paper [code: reconstruct.py#L387]
  [code: reconstruct.py#L94]. Implement one or the other, not both.
- **PSHuman: paper vs its own code, optimisation steps.** The paper's §4 gives
  700 / 100 / 100 steps for alignment, geometry and texture, while §3.2 says
  remeshing takes 700 steps [arxiv:2409.10141 §4] [arxiv:2409.10141 §3.2]. The
  code runs 100 alignment steps and 700 remeshing steps
  [code: reconstruct.py#L276] [code: remesh.yaml#L12].
- **Cross-paper: One-2-3-45++ against SV3D on 3D quality.** The One-2-3-45++
  table ranks it first on F-Score over 1,030 GSO shapes, ahead of Zero123 XL at
  91.6 [arxiv:2311.07885 tab. 1]. SV3D's 3D table, on 50 GSO objects, gives
  One-2-3-45++ a Chamfer distance of 0.054 and 3D IoU of 0.406, behind Stable
  Zero123 at 0.039 and 0.550 [arxiv:2403.12008 §4]. The metrics and object sets
  differ, so the two rankings are not reconciled here.
- **Timings across the variants.** One-2-3-45++ reports 60 s [arxiv:2311.07885 tab. 1],
  PSHuman about one minute [arxiv:2409.10141 §1], SV3D about 40 s per orbit
  [arxiv:2403.12008 app. D], Make-It-3D about 2 hours of training
  [arxiv:2303.14184 §4.1], and RealFusion about 45 minutes of reconstruction on
  one V100 [arxiv:2302.10663 app. A]. The hardware is not stated for the first
  three, or differs, and the last two measure different things. The times
  cannot be compared.
- **RealFusion: paper vs its own code.** The paper places the coarse-to-fine
  switch halfway through training, about step 2500 of 5000
  [arxiv:2302.10663 §3.2] [arxiv:2302.10663 app. A]. The code lifts the masking
  only after step 3000 [code: nerf/options.py#L91-L92]. The paper's front band
  for prior cameras is ±30° of azimuth, while the code uses absolute azimuth
  [arxiv:2302.10663 app. A] [code: nerf/provider_synthetic.py#L47-L64]. The
  paper runs camera replacement every tenth step, while the code runs it on nine
  of every ten [arxiv:2302.10663 app. A] [code: nerf/trainer.py#L367-L368].

## Papers

- [One-2-3-45++ (arXiv:2311.07885)](/papers/2311.07885): joint six-view tile from a fine-tuned Stable Diffusion 2, two sparse-volume 3D diffusion stages and a TensoRF texture pass.
- [PSHuman (arXiv:2409.10141)](/papers/2409.10141): body and face multi-view diffusion conditioned on SMPL-X renders, then SMPL-X-initialised remeshing for full-body humans.
- [SV3D (arXiv:2403.12008)](/papers/2403.12008): Stable Video Diffusion fine-tuned to render a camera-steered orbit, then a NeRF and DMTet fit with masked score distillation.
- [Make-It-3D (arXiv:2303.14184)](/papers/2303.14184): per-object NeRF optimised with a diffusion prior, then point-cloud refinement with a learned renderer.
- [RealFusion (arXiv:2302.10663)](/papers/2302.10663): one RGBA photo, a radiance field fitted with score distillation on random views, prompted by a token learned from augmented copies of the photo.