One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization

In plain words

Many ways of turning a photo into a 3D object spend minutes or hours adjusting a 3D model until its renders match the picture. One-2-3-45 skips that loop. It first asks an image model to imagine the object from several other angles, then hands those guesses to a network that builds the shape in one pass.

The imagined views do not agree perfectly with each other, so the reconstruction network is trained to cope with that. The method also works out how high the camera was when the photo was taken, because the reconstruction needs that angle. The output is a textured mesh, and the authors report that the whole run takes about 45 seconds on one A100 GPU. The same pipeline can start from a text-to-image model instead of a photo.

TL;DR

• Single image in, full 360-degree textured mesh out, in one feed-forward pass with no per-shape optimisation §1 §3.
• Zero123 makes 8 anchor views, each with 4 nearby views, so 32 source views in total §4.1. The input's elevation is estimated from 4 nearby views §3.4.
• The reconstruction module is an SDF cost-volume network based on SparseNeuS. It is trained with two-stage source-view selection and with a mix of ground-truth and predicted views §3.3.
• Author-reported best F-score (83.5 average) and about 45 s per shape on an A100 tab. 1 §4.2.
• Official code is Apache-2.0. Its released training config turns off the depth loss that the paper weights at 1. See Paper vs code before reproducing training.

Problem

Per-shape optimisation with a 2D diffusion prior takes tens of thousands of iterations and often tens of minutes per shape. It is memory-heavy at high resolution, often gives 3D-inconsistent shapes such as two faces or the "Janus" artefact, and often gives poor geometry from density fields §1.

The paper's own test shows why simply lifting the views with a standard reconstruction method does not work. TensoRF (NeRF-based) and NeuS (SDF-based), given 32 Zero123 views with ground-truth poses, produce distortions and floaters §3.2 fig. 3. Zero123's views are close in silhouette and CLIP similarity but not in pixel appearance. PSNR is lowest for large relative poses and for unusual targets such as top and bottom views §3.2 fig. 4.

Assumptions the method relies on:

• The input photo's camera elevation is unknown and must be estimated, since the reconstruction needs absolute camera poses §3.4.
• The object must be separable from its background. The authors use SAM with a box prompt §4.1 app. training.
• Zero123 is treated as a fixed prior and is not fine-tuned §4.1.

Method

The pipeline has three modules: multi-view synthesis with Zero123, elevation estimation of the input view, and 3D reconstruction §3 fig. 2.

Zero123 view synthesis §3.1

Zero123 is a Stable Diffusion model fine-tuned to render an object from a new camera, given one view and a relative camera change. The change is given as spherical offsets (Δθ, Δφ, Δr) for polar angle, azimuth and radius §3.1 fig. 2. One-2-3-45 uses it frozen §4.1.

Source views §3.3 §4.1

For each input image:

• 8 anchor views are rendered around the object. The paper says these 8 camera poses are placed uniformly on the sphere §4.1. The released code does this on two rings instead utils.py; see Paper vs code.
• Each anchor gets 4 local views 10 degrees apart in elevation and azimuth §4.1 app. elevation. That gives 4 × 8 = 32 source views.

The released code's order is: 4 anchors at the zero offset, then 4 local views around anchor 0 (used for elevation), then the elevation estimate, then the second group of anchors, then the remaining local views run.py.

Elevation estimation §3.4 app. elevation

1. Match features between each of the 6 pairs of the 4 nearby views, using LoFTR app. elevation elev_est_api.py.
2. For each candidate elevation, build the 4 camera poses from it and the relative offsets. Triangulate matches from two views, reproject into a third, and average the L1 reprojection error over all triplets [app. elevation] elev_est_api.py.
3. Search coarsely over candidates 10 degrees apart, then finely at 1-degree steps within ±10 degrees of the best coarse value app. elevation elev_est_api.py.
4. Keep the candidate with the smallest error. The estimate takes under one second per shape app. elevation.

The paper says the input view is taken at radius 1.2 and azimuth 0 when building these poses app. elevation. The released code uses azimuth 30 for the input view elev_est_api.py.

Reconstruction module §3.3 fig. 2

The module takes the 32 posed source images and returns a signed distance field (SDF) and a colour field in one pass:

• a 2D feature network extracts features from each source image §3.3;
• a 3D cost volume is built by projecting voxels into every source view and taking the variance of the features there §3.3 one2345_lod_train.conf;
• a sparse 3D CNN turns the cost volume into a geometry volume §3.3;
• an MLP predicts SDF from a point and its interpolated features, and a second MLP predicts blending weights over the source views for colour §3.3;
• an SDF-based volume rendering step, following NeuS, renders colour and depth §3.3.

The mesh is extracted with marching cubes from the SDF. Vertex colours are queried as in NeuS §3.3.

Training §3.3 §4.1 app. training

• Training shapes come from the LVIS subset of Objaverse: 46k models in 1,156 categories. Ground-truth RGB and depth are rendered with BlenderProc §4.1 app. training.
• For each shape, render n = 8 ground-truth views on the sphere. For each of them Zero123 predicts 4 nearby views, so 4n = 32 predicted views are the input §3.3. The data loader draws the source views from these 32 files One2345_train.py.
• Each training sample picks one of the 8 ground-truth views at random as the target. The target's RGB and depth supervise the output §3.3 One2345_train.py.
• The loss is the weighted sum in eq. 1. The paper gives its weights in app. training.
• At inference the 8 ground-truth views are replaced by Zero123 predictions, and no depth input is needed §3.3. The paper says that mixing ground-truth and predicted views during training is essential §4.3.

# One-2-3-45 inference, in the order the released code runs it
# Input:  image (H, W, 3) containing one object
# Output: textured mesh
def image_to_mesh(image):
    # Remove the background with SAM and resize to 256 px  [§4.1]
    object_image = preprocess(image)                  # (256, 256, 3)

    # Four anchor views at the zero-offset pose  [code: run.py#L24-L25]
    anchors = zero123_views(object_image, anchor_offsets[:4])

    # Four local views around anchor 0, 10 degrees apart  [§4.1]
    # [code: run.py#L29-L30]
    local_views_0 = zero123_views(anchors[0], local_offsets)

    # Estimate the input elevation from those four views  [§3.4]
    # [app. elevation]; the released code falls back to 90 degrees
    elevation = estimate_elevation(local_views_0)

    # Second anchor group depends on the estimate  [code: run.py#L40-L44]
    anchors += zero123_views(object_image, second_group(elevation))

    # Local views for the remaining anchors, 4 each  [code: run.py#L51-L54]
    local_views = [zero123_views(a, local_offsets) for a in anchors[1:]]
    source_views = [local_views_0] + local_views     # 32 views of (3, H, W)

    # Poses of all 32 source views from the elevation  [§3.4]
    # [code: utils/utils.py#L106-L145]
    poses = source_poses(elevation)                  # (32, 4, 4)

    # One feed-forward reconstruction and mesh extraction  [§3.3]
    sdf, colour = reconstruct(source_views, poses)
    return marching_cubes(sdf), colour

# One training step of the reconstruction module  [§3.3] [eq. 1]
# Input:  one shape with n = 8 ground-truth renders, depths and poses
# Output: scalar loss
def train_step(shape):
    # Choose one ground-truth view as the target  [§3.3]
    target = random_choice(shape.renders)            # (3, H, W)
    # The 32 Zero123 predictions and their ground-truth poses are the input
    # [§3.3]
    sources = shape.predicted_views                  # (32, 3, H, W)
    rgb_pred, depth_pred = reconstruct(sources, shape.poses, target)
    # Weighted sum of colour, depth, Eikonal and sparsity losses  [eq. 1]
    # [app. training]
    loss = rgb_l1(rgb_pred, target.rgb)
    loss += 1.0 * depth_l1(depth_pred, target.depth)
    loss += 0.1 * eikonal(sdf_gradient)
    loss += 0.02 * sparsity(sdf)
    return loss

Loss equation as stated in the paper eq. 1, app. training:

\mathcal{L} = \mathcal{L}_{rgb} + \lambda_0\mathcal{L}_{depth} + \lambda_1\mathcal{L}_{eikonal} + \lambda_2\mathcal{L}_{sparsity} eq. 1

Hyperparameters

Values marked "paper" come from the text, and values marked "code" come from the released files. Where the two disagree, see Paper vs code.

Results

Author-reported. Setup: 20 random shapes from Objaverse and 20 from Google Scanned Objects (GSO), one rendered input image per shape. Runtime is measured on an A100 GPU. Baselines were run with their official codebases §4.2 app. eval. The F-score uses a 0.05 threshold and CLIP similarity compares renders to the input §4.2.

Source: tab. 1 §4.2. Bold and underline in the paper mark the best and second-best values per column.

• The paper says it beats every baseline on F-score and is second to Shap-E on CLIP similarity §4.2. The authors say CLIP similarity is sensitive to colour and weak on leg or handle counts §4.2.
• Time: the reconstruction module takes about 5 s. Zero123 takes about 1 s per image on an A100, and the paper attributes the rest of the 45 s to Zero123 §4.2.
• Elevation: the estimator's median error is 5.4 degrees and mean error 9.7 degrees on 1,700 random renders fig. 7 §4.3.
• Ablations are shown only as figures fig. 8 fig. 9 fig. 10. Removing the 2-stage source-view selection makes training fail. Using only 8 views without the 4 nearby views misses fine geometry. Dropping the depth loss misses fine geometry. Training only on ground-truth views fails at inference with missing regions. The number of source views matters little once the module is retrained §4.3 fig. 8.
• Text-to-3D: a text-to-image model generates the input, which is then lifted to 3D. No numbers are given, only comparisons in figures §4.4 fig. 11 app. text-to-3D.

Paper vs code

Checked at commit 9d5e3a07ce3a830bf4178cada6670996bf94d037, the One-2-3-45/One-2-3-45 repository, Apache-2.0 LICENSE. The paper links the project site one-2-3-45.com, which redirects to one-2-3-45.github.io. I did not check whether that site links this repository. The repository says it is the official code for the paper. The repository was cloned with the Git LFS demo files failing to download, so only the non-LFS files are checked out; all files cited below are present.

• Depth loss weight. Paper: λ0 = 1 app. training and depth supervision is called essential §4.3. Released training config: depth_loss_weight = 0.0 one2345_lod_train.conf. If the key were absent, the trainer would use 1.0 trainer_generic.py. The paper does not say which setting produced its model.
• Training length. Paper: 300k iterations app. training. Released config: end_iter = 200000 one2345_lod_train.conf. The released demo checkpoint is named for iteration 215000 download_ckpt.py.
• Number of training views. Paper: 4n = 32 predicted views §3.3 §4.1. Released config: nviews = 5 one2345_lod_train.conf. The training data loader reads 32 local views One2345_train.py and I found no use of nviews in that file.
• Anchor layout. Paper: 8 poses uniformly placed on the sphere §4.1. Released code: two rings of 4, one at the estimated elevation and one 30 degrees above or below it, with the ring chosen by the estimated angle utils.py. The switch is at 75 degrees utils.py.
• Elevation search pose. Paper: input at azimuth 0, radius 1.2 [app. elevation]. Code: input at azimuth 30, nearby views offset ±10 degrees in elevation and azimuth elev_est_api.py. The paper says azimuth and radius can be chosen freely, so this changes the result only by a rotation and scale.
• Failure of elevation estimation. Paper: silent. Code: any exception sets the polar angle to 90 degrees and continues run.py.
• Reconstruction levels. Paper: one cost-volume module §3.3. Released config: a single level, 96³ voxels, with a comment saying "only use lod0" one2345_lod_train.conf [code: reconstruction/confs/one2345_lod_train.conf#L66]. A second, finer level is present in the config but disabled.
• Matching paper values. λ1 = 0.1 and λ2 = 0.02 match the paper one2345_lod_train.conf app. training.
• Evaluation code. Not checked. The repository has a render/ folder with an evaluation script that I did not read.

No other discrepancies found in the parts checked.

Pitfalls

• Depth weight. The paper's λ0 = 1 and the released config's 0.0 differ. Reproducing the paper means using the code's default weight, not the released config's value, and this is a guess that no source confirms.
• Elevation drives everything. Errors of ±30 degrees in the input elevation distort the mesh §4.3 fig. 10. The elevation search is also the most fragile step, since the released code silently falls back to 90 degrees when matching fails run.py.
• Inconsistent upstream views. Zero123 is the source of most failures: the back of a fox, a banana with ambiguous structure app. failure fig. 15. Artefacts on the back of the mesh are reported as well app. failure.
• Runtime breakdown. The paper gives 45 s total, about 5 s for reconstruction, and about 1 s per Zero123 image §4.2. It does not state how many Zero123 calls the 45 s includes, so the breakdown cannot be rebuilt from the text.
• Evaluation alignment. Baselines are aligned to ground truth with a search over scale and rotation, followed by ICP, keeping the best inlier count app. eval. Scores therefore depend on the ground-truth mesh. The F-score threshold units are not stated.
• Author-reported only. Single runs, no error bars, and the baselines were run with their official code app. eval.
• Resources. Training took about 6 days on 2 A10 GPUs app. training. The demo needs an NVIDIA GPU with at least 18 GB README.md. The install needs torchsparse v1.4.0 README.md. The LoFTR weights are downloaded from Google Drive at first use elev_est_api.py.
• Figure and equation numbers. Numbers in this card are counted from the source order of captioned figures, tables and equations. Check them against the compiled PDF. The appendix has no \appendix command, so its numbering is not stated. It is cited by subsection title.
• Training data. The released README describes the training images as a CC-BY subset of Objaverse-LVIS README.md. The paper says only "LVIS subset" §4.1.

Lineage

• Builds on: Zero123, a view-conditioned Stable Diffusion model §3.1; SparseNeuS, the generalisable SDF cost-volume network whose training recipe is extended §3.3; NeuS, for SDF-based rendering and colour queries §3.3; LoFTR, for elevation matching app. elevation. No cards exist yet for Zero123 or SparseNeuS.
• Compared against: Point-E, Shap-E, Zero123 with Stable DreamFusion, 3DFuse, RealFusion §4.2 tab. 1.
• Superseded by: not stated in this card.