Text-to-3D generation

Overview

The input is a text prompt; the output is a 3D asset that looks like the prompt when viewed from any side. No paired text and 3D data is needed for the two variants here. Each method starts from a simple 3D representation, renders it from random cameras through a differentiable renderer, scores the renders with a pretrained model, and pushes the gradients back into the representation 2203.13333 §3 2309.16585 §4.1.

The shared difficulties are the same across variants. Optimising only for the text can produce tangled geometry, because the scoring model can be satisfied by painting details onto a flat surface instead of changing the shape 2203.13333 §1. The other common failure is multiple faces: a second face appears on the back of a head, for example 2309.16585 §1. Each variant addresses these with its own constraints (see Variants).

The variants differ on three axes: the representation (a mesh with texture, or a cloud of Gaussians), the scoring signal (CLIP similarity, or score distillation from a diffusion model), and any explicit 3D prior.

Variants

CLIP-Mesh: mesh optimisation with CLIP

The representation is a Loop subdivision surface controlled by a coarse mesh, plus a texture map and a normal map. The loss is the CLIP similarity between random rendered views and the prompt, a diffusion-prior term, and a Laplacian smoothness term 2203.13333 §3 2203.13333 eq. 4. The paper adds random backgrounds, camera offsets and lighting changes so the shape must hold up from every side 2203.13333 §3 Random Augmentations. Card: CLIP-Mesh.

The authors report CLIP R-precision above Dreamfields in all four pairings of generation and evaluation CLIP models, for example 96.7 against 93.5 with ViT-B/16 on both sides 2203.13333 tab. 1. The evaluation uses 153 captions and a held-out view, and the prior loss is left out of those numbers 2203.13333 §4.3. The authors report about 50 minutes per shape on one 16 GB GPU, against more than 24 hours on four A100s for Dreamfields 2203.13333 §4.1.

GSGEN: 3D Gaussians with score distillation and a 3D prior

The representation is a set of anisotropic 3D Gaussians, rendered by splatting 2309.16585 §3.2. The scoring signal is score distillation sampling (SDS) from Stable Diffusion v1.5 2309.16585 §5.1. An initial cloud from the Point-E text-to-point-cloud model sets the rough shape, and a 3D SDS term on the Gaussian centres keeps that shape during the geometry stage 2309.16585 §4.1 2309.16585 §4.3. An appearance stage then optimises colour and opacity with 2D SDS only, adding Gaussians where gaps remain and removing faint ones 2309.16585 §4.2. Card: GSGEN.

Results are visual only. The paper reports fewer multi-face artefacts and finer detail than Magic3D and Fantasia3D on the prompts it shows, and fewer multi-face failures than DreamFusion, ProlificDreamer and Fantasia3D. It reports about 40 minutes per asset on one GPU 2309.16585 §5.2 2309.16585 app. A. The multi-face problem is reduced but not removed 2309.16585 §6.

When to use which

Implementation sketch

The loop is common to both variants. Only the representation, the scoring term and the schedule change between them. The sketch follows the common structure in the cards and is not a released implementation.

# Common optimisation loop for text-to-3D  [arxiv:2203.13333 §3]
#                                          [arxiv:2309.16585 §4]
def train_step(representation, prompt, step):
    # Render from a random camera, background and lighting  [arxiv:2203.13333 §3]
    camera = sample_camera()
    image = render(representation, camera)
    # Score the render against the prompt. CLIP similarity for CLIP-Mesh;
    # SDS from a diffusion model for GSGEN  [arxiv:2203.13333 eq. 1]
    #                                       [arxiv:2309.16585 §4.1]
    loss = guidance_loss(image, prompt)
    # Optional 3D prior: a point-cloud or diffusion prior on the shape
    # GSGEN applies it to Gaussian centres only in the geometry stage
    # [arxiv:2309.16585 §4.1]
    loss = loss + prior_weight(step) * prior_loss(representation, prompt)
    # Shape regulariser on the mesh or the Gaussian positions  [arxiv:2203.13333 eq. 2]
    loss = loss + regulariser_weight(step) * shape_penalty(representation)
    # Update and, for Gaussians, densify or prune on a schedule  [arxiv:2309.16585 §4.2]
    update(representation, loss)
    if schedule_hits(step):
        densify_or_prune(representation)
    return loss

Where the variants differ: CLIP-Mesh has no densification step and its regulariser is a Laplacian on the surface 2203.13333 eq. 2. GSGEN uses a penalty on position norms and an opacity term instead, plus densification and pruning on a schedule 2309.16585 eq. 4 2309.16585 §4.2. Both papers describe their weight schedules only partly, so the schedules must be set from the stated values, not assumed.

Contradictions

• CLIP-Mesh, elevation range. The method section gives a range from 0° to 100° 2203.13333 §3 Method. The evaluation section says training is limited to 30° 2203.13333 §4.3. Neither is settled by the paper.
• CLIP-Mesh, prior in the ablation table. The table row with the prior loss is headed by a B/16 generation model, but the text says the prior was not used with B/16 generation 2203.13333 tab. 2 2203.13333 §4.3.
• GSGEN, mean penalty target. The mean penalty is described as pulling positions towards the origin in the equation and towards the geometry-stage positions in the text 2309.16585 eq. 4 2309.16585 §4.2.
• Timing claims across papers. CLIP-Mesh reports about 50 minutes against more than 24 hours for Dreamfields 2203.13333 §4.1. GSGEN reports about 40 minutes, on par with Magic3D and Fantasia3D 2309.16585 §5.2. The baselines, hardware and prompts differ, so these numbers do not contradict each other, but they cannot be compared directly.

Papers

• CLIP-Mesh: zero-shot textured mesh from text, using a Loop subdivision surface, CLIP similarity and a diffusion prior.
• Text-to-3D using Gaussian Splatting (GSGEN): 3D Gaussians with SDS and a Point-E prior, in a geometry and an appearance stage.