One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion

In plain words

Making a 3D object from a single photo is hard, because a photo shows only one side of an object. Earlier tools either took a long time per object or produced views that did not agree with each other, so the 3D shape came out distorted.

This method works in two stages. First, an image model draws the object from several fixed viewpoints, and it draws them together in one picture so the views can match. Second, a 3D model reads those views and builds a solid shape, which is then turned into a surface with a colour texture. A final step adjusts the colours so the surface matches the drawn views.

The authors report that this works in about a minute and stays close to the input photo. The comparisons use a set of 1,030 3D objects and a user study, and the paper reports them itself.

TL;DR

• Single RGB image in, textured mesh out, in about 60 s (author-reported) tab. 1.
• Stage 1 generates six consistent views by fine-tuning Stable Diffusion 2 on a 3×2 tile §3.1 fig. 3.
• Stage 2 is two diffusion networks over volumes (occupancy, then sparse SDF and colour), conditioned on multi-view features §3.2 fig. 4.
• Best reported F-Score (93.6) and CLIP similarity (81.0) on the 1,030-shape GSO set tab. 1.
• The official repository holds only a README, so the code cannot be checked against the paper.

Problem

Optimisation methods such as DreamFusion take tens of minutes to hours for one shape §1. The feed-forward One-2-3-45 is fast, but its multi-view predictions from Zero123 are inconsistent across views, which limits the reconstruction §1. Zero123 predicts each view on its own, without communication between views §3.1. Conditioning on views also needs the input's elevation, and an error there propagates §3.1. This paper targets speed and fidelity to the input image at once [Abstract].

Method

Overview. Three modules run in sequence. A fine-tuned 2D diffusion model produces six views tiled into one image. A multi-view conditioned 3D diffusion model produces a coarse-to-fine volume. A texture refinement step fits a colour field to the generated views §3 fig. 2.

Consistent multi-view generation §3.1

1. Six target views are tiled into one image in a 3×2 layout, so the views are generated in a single diffusion pass §3.1 fig. 3.
2. The tile is generated by a fine-tuned Stable Diffusion 2 conditioned on the input image §3.1.
3. The input image enters through a local condition (reference attention: the reference image's self-attention keys and values are appended to the denoising UNet's attention layers) and a global condition (CLIP image embedding, multiplied by learnable weights, replacing the text tokens) §3.1.
4. The noise schedule is switched from the original scaled-linear one to a linear one §3.1.
5. Poses use fixed absolute elevations and relative azimuths, so no input elevation has to be estimated §3.1.

# Consistent multi-view generation  [§3.1] [fig. 3]
# Input:  reference image (H, W, 3)
# Output: tile of six views (3H', 2W', 3)

def generate_multi_view(reference_image):
    # Fixed poses: elevations alternate 30 and -20 degrees,
    # azimuths start at 30 and step by 60 degrees  [§3.1]
    poses = six_fixed_poses()                      # six (elevation, azimuth)
    # Local condition: reference attention; global: CLIP embedding  [§3.1]
    local_features = reference_unet(reference_image)
    global_embedding = clip_image_embedding(reference_image)
    # Denoise the tiled composite with the fine-tuned Stable Diffusion 2  [§3.1]
    tile = sample_tiled_views(
        local_features,
        global_embedding,
        poses,
    )
    return tile

3D volume representations §3.2

1. The textured shape is two discrete volumes: a signed distance function (SDF) volume and a colour volume §3.2.
2. The SDF volume gives, per cell, the signed distance from the cell centre to the nearest surface. The colour volume gives the colour of the closest surface point §3.2.
3. An occupancy volume is derived from the SDF by thresholding its absolute value against a predefined threshold §3.2.

Two-stage diffusion §3.2

1. Stage 1 denoises a low-resolution full occupancy volume F of size 64³ that approximates the shell §3.2.
2. Each occupied voxel from stage 1 is subdivided into 8 smaller voxels to form a sparse grid of 128³ §3.2.
3. Stage 2 denoises a sparse volume S of size 128³ × 4 that holds the SDF and colour of voxels in the occupied area §3.2.
4. Both stages use a denoising loss that predicts the clean volume x0 from the noisy volume, the time and the condition c §3.2 (unnumbered equation).
5. Stage 1 uses 3D convolution; stage 2 uses 3D sparse convolution §3.2.

# Denoising loss for one stage  [§3.2] (unnumbered equation)
# x0: clean volume (F or S), condition: multi-view features C
# Output: scalar loss

def denoising_loss(model, x0, condition):
    # Sample Gaussian noise and a continuous time in (0, 1)  [§3.2]
    noise = random_normal_like(x0)
    time = uniform(0.0, 1.0)
    # Noise the clean volume; the noising formula is not stated  [§3.2]
    x_t = add_noise(x0, noise, time)
    # Predict the clean volume and compare with the target  [§3.2]
    predicted_x0 = model(x_t, time, condition)
    return squared_l2(predicted_x0 - x0)

Multi-view condition §3.2 fig. 4

1. A pretrained DINOv2 backbone extracts local patch features for each of the m views §3.2.
2. Each 3D voxel is projected into every view with the known camera poses, and the patch features it lands on are gathered §3.2.
3. A shared-weight MLP processes the m features per voxel, then max pooling over views gives one feature per voxel. This forms the condition volume C §3.2.
4. A 3D convolution network turns C into volumes at the UNet's resolutions; these are concatenated with the UNet's own feature volumes §3.2.
5. For stage 2, the condition is built as a sparse volume and processed with sparse convolution §3.2.
6. The colour stage also receives 2D pixel-wise projected colours at the last UNet layer, and the CLIP feature of the input image is a global condition §3.2.

# Multi-view local condition  [§3.2] [fig. 4]
# Input:  views (m, H, W, 3), poses (m, 4, 4), voxel centres (V, 3)
# Output: condition volume (V, C)

def condition_volume(views, poses, voxel_centres):
    # Local 2D patch features per view from DINOv2  [§3.2]
    patch_features = dinov2_patch_features(views)  # (m, P, C)
    # Project each voxel into each view and gather its feature  [§3.2]
    gathered = gather_by_projection(
        patch_features,
        voxel_centres,
        poses,
    )                                              # (m, V, C)
    # Shared MLP per view, then max pooling over the views  [§3.2]
    per_view = shared_mlp(gathered)                # (m, V, C')
    return max_over_views(per_view)                # (V, C')

Inference §3.2

1. A 64³ grid is initialised with Gaussian noise and denoised by the stage 1 network §3.2.
2. Each predicted occupied voxel is split into 8 smaller voxels, forming the sparse grid §3.2.
3. The sparse grid is initialised with Gaussian noise and denoised by the stage 2 network, giving SDF and colour per voxel §3.2.
4. Marching Cubes extracts a textured mesh §3.2.

Texture refinement §3.3

1. The mesh geometry is fixed §3.3.
2. A colour field represented by TensoRF is optimised; each iteration rasterises the mesh and queries the colour network §3.3.
3. The loss is L2 against the consistent multi-view images §3.3.
4. The optimised colour field is baked onto the mesh, with the surface normal used as the viewing direction §3.3.

# Texture refinement with a TensoRF colour field  [§3.3]
# Input:  mesh with fixed geometry, generated views with poses
# Output: textured mesh

def refine_texture(mesh, views, poses):
    # Optimise only the colour field; the geometry stays fixed  [§3.3]
    color_field = init_tensorf_color_field()
    for step in refinement_steps():               # count not stated
        # Rasterise the mesh and query the colour network  [§3.3]
        rendered = rasterize_and_query(mesh, color_field, poses)
        # Compare with the generated views using an L2 loss  [§3.3]
        loss = l2_loss(rendered, views)
        update(color_field, loss)
    # Bake the colour onto the mesh; the normal is the viewing direction  [§3.3]
    return bake_color(mesh, color_field)

Hyperparameters

Results

Author-reported. Setup: GSO dataset, 1,030 shapes, one frontal input per shape, F-Score and CLIP similarity §4.1 tab. 1. Text-to-3D uses 50 DreamFusion prompts §4.2 tab. 2. Ablations use the full GSO set §4.3.

Image-to-3D tab. 1

User study: 53 participants, 2,385 pairs, 45 shapes each §4.1 fig. 6. One-2-3-45++ beats One-2-3-45 in 92% of pairs §4.1 fig. 6.

Text-to-3D tab. 2

User study: 53 participants, 1,590 pairs, 30 pairs each §4.2. The paper reports 70% preference over MVDream §4.2.

Module ablation tab. 3 (full GSO set)

3D diffusion ablation tab. 4 (3D IoU of stage 1 occupancy shell)

Multi-view generation tab. 5 (GSO, single input image)

Time breakdown: the paper gives no per-stage timing. Tab. 3 implies texture refinement adds about 40 s (60 s minus 20 s) tab. 3.

Paper vs code

The official repository, SUDO-AI-3D/One2345plus, is linked from the project page named in the abstract [abstract]. At code_commit it contains only README.md README.md. There is no model, training, sampling or configuration code, and no LICENSE file. The repository's README repeats that the model "can be trained with only 8 A100 GPUs" README.md; this matches the paper's 8×A100 statement §3.2. No other discrepancies can be checked, because no implementation is published.

The repository was found through the project page link. A GitHub search of the SUDO-AI-3D organisation returned only this repository and the project page's own repository, so no other official code exists. No third-party reimplementation was used.

Pitfalls

• No implementation is published. Network sizes, learning rates, optimisers, batch sizes, sampler and step counts, guidance scale, occupancy threshold, sparse convolution details, TensoRF settings and inference hardware are all not stated in the paper or the repository. An implementer will have to choose these. Treat the results as reproducible only in outline.
• Supplementary material is missing. The paper defers details to the supplementary material in several places §3.2. The supplementary is commented out of the source we have (X_suppl.tex is not included in main.tex), so those details are unavailable here.
• Equation numbering. The denoising loss is an unnumbered equation §3.2. Cite it by section, not by equation number.
• Figure numbering. Figures are cited here by their order in the compiled paper: Fig. 1 teaser, Fig. 2 pipeline, Fig. 3 tiling, Fig. 4 multi-view condition, Fig. 5 image-to-3D results, Fig. 6 user study, Fig. 7 qualitative results, Fig. 8 text-to-3D results §1 §2 §3 §4.
• Text-to-3D pipeline is not described. The paper evaluates text input §4.2 tab. 2, but does not say how the text becomes an input image for the image pipeline. Treat text-to-3D numbers as "not reproducible from this card".
• Noise schedule. Fine-tuning switches to a linear schedule from the original scaled-linear one §3.1. Get this right when porting the 2D stage. The 3D diffusion schedule is not stated.
• Training-inference gap. The 3D network is trained on ground-truth renderings, not on generated views §3.2. Ablation rows d and e show that training on predicted views lowers 3D IoU (41.9 vs 44.1) tab. 4. Projection perturbation during training raises it from 44.1 to 45.1 tab. 4. Magnitudes of perturbation are not stated.
• Pose convention. Poses are given as elevation and azimuth. The camera convention (world frame, distance, field of view, and how the tile is laid out in pose order) is not stated; the azimuth order is assumed to follow the layout.
• Projection detail. How a projected voxel position maps to a patch feature (interpolation, patch size) is not stated §3.2.
• Colour path. The colour stage concatenates 2D projected pixel colours at the last UNet layer §3.2. How colours are sampled and normalised is not stated.
• Claim vs table. The multi-view section says the method "surpasses current methodologies in PSNR, LPIPS, and foreground mask IoU" §4.3. Tab. 5 shows SyncDreamer has lower LPIPS (0.095 vs 0.110) and higher mask IoU (0.894 vs 0.878). The comparison also uses different target elevations per method §4.3 tab. 5.
• Metric definitions. The F-Score distance threshold is not stated. The alignment step ("linear search and ICP") has no parameters §4.1. CLIP-Sim averages over 24 views per mesh §4.1.
• Dataset caveat. GSO shapes are said to be unseen by all methods "to the best of our knowledge" §4.1. Objaverse is the training data §3.1 §3.2; overlap is not checked by the authors.
• Cost. Stage 1 fine-tuning took 16 GPUs for about 10 days §3.1. Each 3D stage took 8×A100 for about 10 days §3.2. Inference takes about 60 s tab. 1, and the hardware is not stated.
• Naming. Table 3's "Reconstruction" column means the 3D diffusion module (ours or SparseNeuS) tab. 3. The paper also uses "sparse view reconstruction" for the same column.
• Typos in the source. "Obajverse" §3.2 and "v-mode" §3.1 appear in the source. Read them as Objaverse and the v-model.

Lineage

• Builds on: One-2-3-45 (arXiv:2306.16928), which uses Zero123 for views and a generalizable NeRF for reconstruction §1 §3.2. Zero123 (Zero-1-to-3) is the view-conditioned 2D model it is compared to §1.
• Uses: Stable Diffusion 2 for multi-view fine-tuning §3.1; LoRA (arXiv:2106.09685) for the first fine-tuning stage §3.1; DINOv2 features for the 3D condition §3.2; TensoRF for texture refinement §3.3; LAS-Diffusion's coarse-to-fine volumes (arXiv:2305.04461) §3.2.
• Compared against: Zero123 XL, SyncDreamer (arXiv:2309.03453), Wonder3D (arXiv:2310.15008), DreamGaussian, Shap-E, DreamFusion, ProlificDreamer (arXiv:2305.16213), MVDream §4.
• Superseded by: not stated in this wiki.