Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior

In plain words

Give this method one photo of an object and it builds a 3D model that can be viewed from other sides. It works in two passes. The first pass learns a rough 3D shape that matches the photo from the front and still looks plausible from other angles. An image-generation model judges what the unseen sides should look like. The second pass improves the surface. Pixels visible in the photo are copied onto the 3D points that correspond to them, and the hidden parts are filled in by a small learned renderer. The authors report that this beats earlier single-image and text-to-3D methods on their own test set.

TL;DR

Make-It-3D creates a textured 3D object from one image, with no multi-view training data §1. A NeRF is first optimised with a 2D diffusion model as a prior at novel views, plus a reference-view loss, a CLIP-based diffusion loss and a depth prior §3.2. It is then exported to a textured point cloud that is refined with a jointly optimised neural renderer §3.3. Results are author-reported and use their own 400-image benchmark plus DTU [§4.1, tab. 1, tab. 2]. The official code is linked from the project page and contains training code that differs from the paper in several numbers (see Paper vs code).

Problem

Creating 3D from one image is under-determined: the geometry must be inferred and the unseen texture hallucinated §1, §3. Single-view 3D-photo methods (layered or multi-plane images) struggle with fine geometry and large viewpoint changes §1, §2, §4.2. Methods that project an image into a category-specific 3D generator only cover a single object class §1. Text-to-3D score distillation follows the prompt but often misses the details of a given reference image §1, §3.2. A diffusion model used only through a text prompt therefore leaves the reference appearance under-constrained §3.2.

Method

The method has two stages and needs a caption, a foreground mask and a depth estimate of the reference image §3.2, app. B.1.

Preliminaries §3.1. A NeRF rendering is noised and denoised by a frozen diffusion model. Score distillation sampling (SDS) turns the noise residual into a gradient for the NeRF parameters, without back-propagating through the diffusion network. The authors use Stable Diffusion, which works on VAE latents, so the gradient passes through the encoder §3.1:

\nabla_{\theta}\mathcal{L}_{\mathrm{SDS}} = \mathbb{E}_{t,\epsilon}\left[w(t)\,(\epsilon_{\phi}(z_t;y,t)-\epsilon)\,\frac{\partial z}{\partial x}\frac{\partial x}{\partial\theta}\right] eq. 3

Coarse stage §3.2, app. B.1. Steps, each in the authors' words:

1. A captioning model writes a text prompt for the reference image §3.2. The code uses BLIP-2 (OPT-2.7B) and rewrites a few caption phrases main.py.
2. Novel views are sampled around the reference view. The reference view is used with 25% probability and novel views with 75% §4.1. Novel views get a random camera distance and field of view app. B.1.
3. The reference view gets a masked L1 pixel loss, which is also used to separate the object from the background eq. 2 §3.2: \mathcal{L}_{\mathrm{ref}} = \lVert x\odot m - \mathcal{G}_{\theta}(\beta_{\mathrm{ref}})\rVert_1
4. Novel views get SDS with classifier-free guidance. The guided noise estimate is a mix of the conditional and unconditional predictions with weight omega = 10 §4.1, and the timestep is drawn between 200 and 600 §4.1.
5. Novel views also get the diffusion CLIP loss, L_CLIP-D. The rendering is noised, denoised in one step to a clean estimate, and then compared with the reference through CLIP image embeddings §3.2, eq. 4: \mathcal{L}_{\mathrm{CLIP\text{-}D}} = -\mathcal{E}_{\mathrm{CLIP}}(x)\cdot\mathcal{E}_{\mathrm{CLIP}}(\hat{x}_0(\beta,t))
6. SDS and L_CLIP-D are never applied to the same step. L_CLIP-D is used at small timesteps and SDS at large ones. The threshold is T = 400 [§3.2, app. C.1, tab. 3].
7. A depth prior is added. An off-the-shelf monocular depth estimator gives a depth map of the reference image. The NeRF depth at the reference view is pushed to correlate linearly with it, scale-free §3.2, eq. 5: \mathcal{L}_{\mathrm{depth}} = -\frac{\mathrm{Cov}(d(\beta_{\mathrm{ref}}),d)}{\mathrm{Var}(d(\beta_{\mathrm{ref}}))\,\mathrm{Var}(d)}
8. The total loss combines these terms, but the paper gives no weights §3.2. Training is progressive: views start near the reference and the range is widened over time, which allows a full 360 degree reconstruction §3.2.
9. Shading augmentation (Lambertian and normal shading) is applied to the renders §4.1. Training uses a random background colour and renders are composited on white at test time app. B.1.

Representation and optimisation: a multi-scale hash encoding with an occupancy grid for empty-space skipping, and a small MLP that predicts density and colour app. B.1, §4.1. The density starts as a Gaussian blob centred at the scene origin app. B.1.

Refine stage §3.3, app. B.2. Steps:

1. Export the coarse NeRF at the reference view as a point cloud. Depth and alpha mask give points, which are coloured with the reference image eq. 6 §3.3: V(\beta_{\mathrm{ref}}) = R_{\mathrm{ref}}K^{-1}\mathcal{P}\big(\mathcal{D}(\beta_{\mathrm{ref}})\ast M(\beta_{\mathrm{ref}})\big)
2. For each other view, project the existing points to find those already observed. Lift only the unobserved points, and colour them with the NeRF render. This avoids duplicates whose colours conflict §3.3.
3. Each point gets a 19-dimensional descriptor. The first three dimensions are its RGB colour and the rest are random §3.3, §4.1.
4. Rasterise the points at K = 3 scales (full, half and quarter resolution), and decode the stack with a U-Net renderer with gated convolutions. The U-Net and descriptors are optimised jointly with the point features [§3.3, eq. 7, app. B.2]. \mathbf{I}(\beta) = \mathcal{R}_{\theta}(I_0(\beta), I_1(\beta), \ldots, I_{K-1}(\beta))
5. The loss is the same kind as in the coarse stage, plus a penalty on large departures of the optimised colours from the initial ones §3.3. Geometry is kept fixed §3.3.

# Coarse stage: a NeRF from one image  [§3.2] [app. B.1] [§4.1]
# Input:  reference x (RGB), mask m, caption y, depth prior d
# Output: NeRF parameters theta

def coarse_stage(x, m, y, d, num_iters):
    for step in range(num_iters):
        # Reference view with probability 0.25, else a novel view [§4.1]
        is_reference = random_uniform() < 0.25
        # Novel views: distance 0.8-1.2, FOV 40-80 deg [app. B.1]
        camera = sample_camera(is_reference)
        # Random background colour during training [app. B.1]
        rgb, depth = render_nerf(theta, camera, random_colour())
        if is_reference:
            # Masked L1 to the reference view [eq. 2]
            loss = l1(x * m, rgb * m)
            # Negative Pearson correlation with the depth prior [eq. 5]
            loss = loss - pearson(depth, d)
        else:
            # Timestep t drawn uniformly from 200 to 600 [§4.1]
            t = random_int(200, 600)
            latent = encode(rgb)
            noisy = add_noise(latent, t)
            # Classifier-free guidance, omega = 10 [§4.1]
            eps_guided = (1 + omega) * eps(noisy, y, t) - omega * eps(noisy, t)
            if t <= 400:
                # CLIP-D: one denoising step, then compare with x [eq. 4]
                x0_hat = denoise_one_step(noisy, t, eps_guided)
                loss = -clip_similarity(decode(x0_hat), x)
            else:
                # SDS gradient w(t) (eps_guided - eps) [eq. 3]
                loss = sds_loss(latent, eps_guided, noise, w(t))
        theta = optimizer_step(theta, loss)
    return theta

# Refine stage: colour the surface and fill hidden texture  [§3.3] [app. B.2]
# Input:  coarse NeRF theta, reference x, mask m, reference depth d_ref
# Output: points P, colours C, descriptors F, neural renderer R

def build_points(theta, x, m, d_ref, cameras):
    # Reference-view points, coloured from the photo [eq. 6]
    points = unproject(d_ref * m, camera_ref)
    colours = sample_pixels(x, points, camera_ref)
    for camera in cameras:
        # Keep only points that no existing point already covers [§3.3]
        covered = project_mask(points, camera)
        new_pixels = visible_and_uncovered(theta, camera, covered)
        # New points take their colour from the NeRF render [§3.3]
        new = unproject(render_nerf_depth(theta, camera), new_pixels, camera)
        points = concat(points, new)
        colours = concat(colours, render_nerf_colour(theta, new, camera))
    return points, colours

def train_refine(points, colours, x, m, cameras, num_iters):
    # 3 colour channels + 16 random channels per point [§3.3] [§4.1]
    descriptors = concat(colours, random_normal(16))
    unet = UNetRenderer()
    for step in range(num_iters):
        camera = sample_camera(step)
        # Rasterise at K = 3 scales, then render with the U-Net [app. B.2]
        maps = [rasterise(points, descriptors, camera, s) for s in (1, 2, 4)]
        image = unet(maps)  # [eq. 7]
        # Coarse-stage style objective plus a colour-drift penalty [§3.3]
        loss = objective(image, x, camera) + colour_drift(colours)
        descriptors, unet = optimizer_step(descriptors, unet, loss)
    return unet

Notes on the pseudocode: the coarse block follows the paper's equations, not the code (see Paper vs code). Loss weights and the per-view objective are not stated in the paper, so the refine objective is left abstract here.

Hyperparameters

Code loss weights (for reference; the paper gives none): reference L1 weight 1e3, depth 1, CLIP 1 main.py; opacity 1e-3, entropy 1, orientation 1e-2, smoothness 1 main.py. Refine stage: masked L1 1000, CLIP 10, background 1e-3, colour regularisers 1e3 and 1e5 utils.py.

Results

Author-reported. Setup: DTU §1, tab. 1 with one reference view, and their 400-image test benchmark of real and Stable Diffusion images with masks, depth and prompts §4.1, tab. 2. Baselines are described in §4.2.

LPIPS is computed at the reference view and the other two metrics at novel views tab. 1 caption. The authors call the enhanced model best on all three §4.2.

Ablation on the coarse-stage losses, on the test benchmark app. C.1, tab. 3:

Applications, shown only as figures: real-scene modelling §5, fig. 9, text-to-3D by first generating a reference image §5, fig. 10, and texture edits on a frozen geometry, such as a tattoo or stylisation §5, fig. 11.

Paper vs code

Code: junshutang/Make-It-3D at commit d5caefb4, linked from the project page [project page]. The repository contains training code for both stages README.md. A Jittor port exists in another repository and is not used here. The code has no LICENSE file.

• Optimiser. Paper: Adam, lr 0.001 for both stages §4.1. Code default is Adan with lr 5e-3 (5 times --lr), weight decay 2e-5 and gradient clip 5.0 main.py. Adam is used only with --optim adam main.py. The refine stage uses Adam, lr 0.001, betas (0.9, 0.99) utils.py.
• Learning-rate schedule. Not stated in the paper. The coarse stage uses a constant schedule for the hash backbone main.py. The refine stage creates a decay schedule utils.py, but step() is never called on it, so the rate stays at 0.001.
• Iterations and resolution. Paper: 5000 coarse at 100x100 and 5000 refine at 800x800 §4.1. Code defaults: 10000 coarse at 128x128 [code: main.py#L37, #L65-L66] and 3000 refine main.py. The README runs the coarse stage as 2000 frontal iterations (phi 135 to 225) followed by 5000 full-circle iterations with --albedo_iters 3500, and then the refine stage [code: README.md]. The second run loads the latest checkpoint from the same workspace by default utils.py, so it continues from the frontal run.
• Camera ranges. Paper: distance 0.8 to 1.2, novel FOV 40 to 80 app. B.1. Code defaults: radius 1.0 to 1.5 main.py, FOV 15 to 25 [code: main.py#L74], reference FOV 20 main.py. The code also samples a polar angle of 70 to 110 degrees main.py, which the paper does not state. The refine stage renders every view with --fov [code: nerf/utils.py#L754].
• Density blob. Paper: mu = 0.2 app. B.1. Code default is 0.1 main.py. The README's long-geometry command passes 0.2 [code: README.md].
• Reference-view loss. Paper: masked L1 eq. 2. Code: L1 on the full composite, where both the rendering and the reference are blended with the same random background colour utils.py, weighted by 1e3 utils.py. The depth loss in code is 1 minus Pearson correlation over foreground pixels only [code: nerf/utils.py#L423-L434, #L556]. The depth map is normalised to 0.1 to 1 with background set to a constant main.py.
• Warm-up. Not in the paper. For the first 400 iterations, and for every reference-view step, no diffusion loss is applied [code: nerf/utils.py#L512, main.py#L50]. Albedo-only shading is used for the first 1000 iterations main.py.
• Shading mix. Non-reference steps use albedo with probability 0.5, textureless shading with 0.1 and Lambertian with 0.4, with ambient ratio 0.1 for the two shaded modes utils.py. The paper names Lambertian and normal shading §4.1.
• Extra CLIP terms. Paper: CLIP-D only, applied to the denoised estimate §3.2, eq. 4. Code: for every non-reference step, a CLIP image loss and a CLIP text loss on the rendered image, each weight 1 utils.py. In the CLIP-D branch there is also a CLIP text loss on the denoised image, weight 10 sd.py. The paper states neither term.
• CLIP-D is skipped for "large" views. Views with phi in 0 to 45 or 315 to 360 degrees are flagged is_large provider.py. For those, the SDS branch is used at every timestep sd.py.
• SDS weight. Paper: "uniform weighting depending on the timestep" §4.1. Code: w(t) = 1 - alpha_bar_t sd.py.
• Geometric regularisers. Paper: sparsity, opacity and smoothness app. B.1. Code: entropy, opacity, orientation and smoothness, with orientation and entropy scaled up 10 times after warm-up and opacity up 10 times for large views utils.py, renderer.py.
• Refine objective. Paper: a coarse-style objective plus a regulariser §3.3. Code: masked L1 with weight 1000 on the reference view [code: nerf/utils.py#L876]. Elsewhere, SDS with guidance scale 5, not 10 [code: nerf/utils.py#L880], a CLIP image loss on the rendering with weight 10 [code: nerf/utils.py#L881], and a VGG relu5_4 contextual loss [code: nerf/utils.py#L810, #L882-L883]. The README notes that the contextual loss is used to sharpen texture. The paper does not mention it. Other terms: background penalty 1e-3 utils.py and colour regularisers 1e3 and 1e5 utils.py.
• Point cloud construction. Code adds Canny edge masking on depth [code: nerf/refine_utils.py#L349-L359, #L386-L395], z-buffer visibility with a 1/H tolerance refine_utils.py, and 33 novel views at theta 80, 90 and 100 degrees provider.py. The paper states none of these.
• Stable Diffusion. Paper: "publicly available" §3.1. Code default is Stable Diffusion 2.0 base main.py.
• Reference preprocessing. Code resizes the reference to 512x512 and erodes the mask with a 5x5 kernel main.py, main.py. The paper does not state this.
• Caption model. Paper: a captioning model, citing BLIP-2 §3.2. Code: BLIP-2 OPT-2.7B main.py. The prompt can be overridden with --text README.md.

Consistent with code: SDS timestep range 200 to 600 sd.py, CLIP-D threshold 400 sd.py, guidance 10 main.py, 19-dimensional descriptors utils.py and the 96-sample NeRF main.py.

Pitfalls

• No loss weights in the paper. The overall loss is described only as a combination §3.2, and the refine-stage weights are absent. The code uses 1e3, 1e5 and others (see Hyperparameters). Treat the code values as the only working set.
• Eq. 4 has no normalisation. The text writes a dot product of CLIP embeddings. The code L2-normalises both embeddings before the product utils.py.
• Eq. 5 is written with Var but described as standard deviation §3.2. Implement it as a Pearson correlation, which is what the code computes with torchmetrics utils.py.
• Eq. 6 is loose. The composition of R, K inverse and P is written without a convention for world or camera frames, depth units or the homogeneous coordinate. The code uses a world-to-camera inversion and a 2-pixel splat radius utils.py, utils.py, so implementers should copy those conventions.
• Point density is not stated. The number of points, the splat size and the stopping rule for point insertion are absent from the paper. The code uses a 2-pixel radius and 8 points per pixel for the z-buffer [code: nerf/utils.py#L765-L766].
• Time-stepping order. SDS at large t and CLIP-D at small t are exclusive per step §3.2, app. C.1. The code applies the same rule but also skips it for large views, and it is ignored in the first 400 iterations [code: nerf/utils.py#L512, nerf/sd.py#L153].
• Hidden costs. Training takes about 2 hours on a 32 GB V100 §4.1. Memory use and the cost of the refine stage's U-Net are not stated.
• Geometry ambiguity. The authors list the Janus problem and over-flat geometry as limitations, with a depth prior that helps only at the reference view app. E. The README warns about "long geometry" and suggests larger reference FOV and blob radius README.md.
• Comparison caveats. Fig. 7 takes DreamFusion images from DreamFusion's website fig. 7 caption. DreamFusion+ is the authors' modified baseline with a reference-view loss §4.2. The paper does not say which subset of the 400 images is used in tab. 2. LPIPS for 3D-Photo is 0 at the reference view tab. 1, which the text does not explain. The CLIP model used for the CLIP score is not stated in the compiled text.
• Benchmark. The authors say they will release the benchmark §4.1. The README lists test alpha data for all paper results as released [code: README.md]. Check the link before relying on it.
• Prompt dependency. Text prompts come from a captioning model, and the caption shapes the SDS signal §3.2. The code lets you replace it with --text README.md.
• Licence. The paper is CC BY 4.0. The code has no LICENSE file, so no reuse rights are granted by the repository. Cite it by path and line only.

Lineage

• Builds on: DreamFusion's score distillation sampling §3.1, Stable Diffusion §3.1, Instant-NGP for the NeRF encoder §4.1, and BLIP-2 for captions §3.2. The reference-view pixel loss follows Yariv et al. for masking §3.2.
• Compared against: DietNeRF, SinNeRF, DreamFusion and DreamFusion+, Point-E and 3D-Photo §4.2.
• Superseded by: not stated in the wiki.