Magic3D: High-Resolution Text-to-3D Content Creation

In plain words

Describing a 3D object in words and getting a usable 3D model back is slow and blurry with existing methods. DreamFusion showed that a picture-making model can act as a judge: it nudges a 3D scene until its views look like plausible images of the prompt. The catch is that this judge only sees small pictures, so the result is soft and the process takes hours.

Magic3D works in two rounds. The first round builds a rough 3D shape quickly, using a compact lookup structure instead of one large network, and judges it with a small-picture model. The second round turns that rough shape into a triangle mesh, the kind of model game engines and 3D tools already use. It is then polished against a judge that sees much larger pictures. The result is sharper geometry and texture, and the authors report it is faster than DreamFusion.

TL;DR

• Two-stage, coarse-to-fine text-to-3D: a hash-grid neural field first, then a textured mesh §4.
• Stage 1 uses a 64x64 diffusion prior; stage 2 uses a 512x512 latent diffusion prior, which is possible because the mesh renders efficiently §4.1.
• Author-reported: about 40 minutes total on 8 A100 GPUs; 61.7% of user-study raters prefer it to DreamFusion §5 tab. 1.
• No official code was found, so every hyperparameter below comes from the paper and its appendix.

Problem

DreamFusion optimizes a NeRF with a text-to-image diffusion prior that works on 64x64 images, so high-resolution geometry and texture are out of reach §3. Its scene model is a large global MLP, which is expensive in compute and memory as resolution grows §3 §4.2. The introduction reports DreamFusion at about 1.5 hours per prompt on average on TPUv4; that figure is quoted from DreamFusion, not measured by the authors on their own hardware §1.

Method

Overview. Magic3D optimizes a parametric scene from a text prompt with score distillation sampling (SDS). The gradient of the SDS loss is back-propagated through a differentiable renderer into the scene parameters §3 eq. 1. The method runs two stages. Stage 1 optimizes a coarse neural field with a base diffusion prior at 64x64. Stage 2 converts that field into a textured mesh and optimizes it with a latent diffusion prior at 512x512 §4 §4.1.

Step 1: diffusion priors §4.1. Stage 1 uses the base diffusion model of eDiff-I, similar to the Imagen base model used by DreamFusion. The loss is defined on 64x64 renders. Stage 2 uses latent diffusion (the publicly available Stable Diffusion model). Gradients reach the 512x512 render through the encoder, and the prior acts on a 64x64 latent §4.1. The increased cost of stage 2 comes from the gradient through the high-resolution render and through the encoder §4.1.

Step 2: scene models §4.2. Stage 1 uses a neural field with a multi- resolution hash grid from Instant-NGP. Two single-layer networks read the grid features: one predicts albedo and density, the other predicts normals. A sparse structure encodes occupancy, and empty space is skipped during rendering §4.2 app. B. Stage 2 uses a textured mesh. The shape is a deformable tetrahedral grid. Each vertex holds a signed distance value and a 3D offset. A surface mesh is extracted with differentiable marching tetrahedra. The texture is a neural colour field evaluated at each rendered pixel's 3D point §4.2.

Step 3: stage 1 optimization §4.3 app. B. The occupancy grid is initialized to 20 and refreshed every 10 iterations, with decay 0.6 per refresh. Normals come from an MLP rather than from finite differences of density §4.3. The background is an environment-map MLP with 16 hidden units and a 10x lower learning rate, so the object, not the background, carries the shape §4.3. The density is biased toward the centre of a sphere before its activation app. B eq. 3. Camera and light are randomly augmented app. B.

Step 4: stage 2 optimization §4.3 app. B. The density field is turned into an SDF by subtracting a non-zero constant (the value is not stated). Vertex SDF values and offsets are optimized with the high-resolution SDS gradient. The texture is initialized from the stage-1 colour field. Each step renders the mesh with a differentiable rasterizer, which tracks the 3D point under each pixel to query the texture. The focal length is raised to zoom in on detail. Background is composited with differentiable antialiasing. An angular regularizer between adjacent faces keeps the surface smooth under the high-variance SDS gradient §4.3.

Step 5: the SDS losses §3 eq. 1 eq. 2 app. B. Stage 1 samples timesteps uniformly from (0, 1) and uses weight w(t) = 1. Stage 2 samples t from (0.02, 0.5) and uses w(t) = σ_t². A t_max in [0.5, 0.7] is said to work well app. B.

# Magic3D coarse-to-fine text-to-3D  [§4]
# Input:  text prompt y
# Output: textured triangle mesh

def text_to_3d(prompt):
    # Stage 1: coarse hash-grid field, SDS on 64x64 renders  [§4.1] [§5]
    coarse = optimize_coarse_field(prompt, steps=5000)

    # Turn the density field into an SDF on a tetrahedral grid  [§4.2] [§4.3]
    mesh = init_tet_mesh_from_density(coarse)

    # Stage 2: SDS on 512x512 renders through the latent prior  [§4.1] [§5]
    mesh = optimize_mesh(mesh, coarse, prompt, steps=3000)
    return mesh

# One stage-1 SDS gradient  [§3] [eq. 1] [app. B]
# Inputs: field (coarse neural field), prompt y, prior (base diffusion model)
# Output: gradient with respect to the field's parameters

def sds_grad_stage1(field, prompt, prior):
    # Render a 64x64 view from a random camera and light  [app. B]
    image = render(field, random_camera(), random_light())
    # Sample a timestep and noise, with w(t) = 1  [app. B] [eq. 1]
    t = uniform(0.0, 1.0)
    noise = normal_like(image)
    # The noise schedule is not stated in this paper  [§3]
    noisy = add_noise(image, noise, t)
    predicted = prior.predict_noise(noisy, prompt, t)
    return (predicted - noise) * d_image_d_params

Extra text-and-image guidance (appendix only). Style and content guidance use an extended classifier-free guidance rule with a text weight and a joint text-and-image weight app. D eq. 4. It is applied only with the eDiff-I prior, because the latent prior does not accept a reference image app. D §6.

Controllability (not part of the main pipeline). DreamBooth fine-tuning of the eDiff-I and latent priors on 11 cat or 4 dog images, then optimization with an identifier token §6. Prompt-based editing fine-tunes a coarse NeRF with a new prompt at high resolution, then optimizes the mesh §6.

Hyperparameters

Results

Author-reported. Setup: 397 text prompts from the DreamFusion gallery; Magic3D is trained on all of them and compared with DreamFusion's published results. Users rate pairs of canonical-view videos on Amazon Mechanical Turk, 3 raters per prompt, 1191 pairwise comparisons in total §5 tab. 1. Hardware: 8 NVIDIA A100 GPUs for both stages §5. The paper reports no FID, CLIP or other automatic metric for the main results.

Paper vs code

No official code. Searches: the paper lists no code link, and its project page (research.nvidia.com/labs/dir/magic3d) links only to a second project page with no code. A GitHub repository search for "magic3d", and the same query restricted to the NVlabs and NVIDIA organisations, returned no official repository. The repositories found under the name are third-party and were not used.

Pitfalls

• Speed comparison is not like for like. The 40-minute figure is measured on 8 A100 GPUs. The 1.5 h DreamFusion figure is quoted from DreamFusion's reported TPUv4 runs, not measured by the authors §1 §5. Treat "2x faster" as author-reported.
• Effective batch size. DreamFusion's global MLP limits it to an effective batch of 8; Magic3D uses 32 app. B.
• Text guidance weight is missing. The paper uses classifier-free guidance, and says DreamFusion relies on large weights §3, but gives no value for its own text-only pipeline. Choose and report one; the stage-2 value is also not stated.
• Background cheating. Because the sparse representation has no contraction, the optimization can learn the object in the environment map. The paper fixes this with a tiny, slowed environment MLP §4.3.
• Normals differ between stages. Stage 1 uses an MLP for normals, which the authors say is acceptable because the render is volumetric. Accurate normals are obtained only in stage 2 with surface rendering §4.3.
• Density bias on the pre-activation. The bias is added before softplus, so the post-activation falls smoothly from softplus(10) to 0 with distance. Adding it after the activation changes the behaviour app. B eq. 3.
• Low-resolution input to the latent prior fails. Rendering the coarse model at 64x64 or 256x256 and upsampling to 512x512 gives worse shapes than the coarse-to-fine approach §5.
• Timestep range matters. The stage-2 range (0.02, 0.5) is chosen empirically, and the authors say t_max affects quality app. B.
• Mesh regularization and offsets. The angular regularizer weight, the SDF offset constant and the tetrahedral grid resolution are not stated. Expect to tune them.
• Style guidance is eDiff-I only. Reference-image guidance cannot use the latent prior, so the high-resolution path does not support it app. D §6.
• Human evaluation only. The comparison is a user study on DreamFusion's published videos §5 tab. 1. It does not test a re-run of DreamFusion.
• Noise schedule and sampler are not restated. This paper does not give the noise schedule or sampler of the eDiff-I and Stable Diffusion priors. An implementation must take them from those models' own sources.

Lineage

• Builds on: DreamFusion (Poole et al.), which the paper modifies: a coarse-to-fine scheme, a hash-grid scene model and a mesh fine stage §3 §4. Also Instant-NGP for the hash grid §4.2 and DMTet/deformable tetrahedral meshes for the mesh stage §4.2.
• Compared against: DreamFusion only §5.
• Superseded by: not known from content in this wiki.