--- kind: concept slug: text-to-3d-generation title: Text-to-3D generation summary: >- Making a 3D object from a text prompt by optimising a 3D representation until pretrained models rate its renders as matching the text. Two variants: CLIP-Mesh (textured mesh and normal map, CLIP and a diffusion prior, zero-shot) and GSGEN (3D Gaussians, Stable Diffusion SDS plus a Point-E prior, about 40 minutes per asset). Results are author-reported. tags: [3d-generation, text-to-3d, differentiable-rendering, clip, score-distillation-sampling] papers: ["2203.13333", "2309.16585"] related: [differentiable-rendering, score-distillation-sampling, gaussian-splatting, diffusion-models, neural-radiance-field, image-to-3d-generation] status: draft updated_at: 2026-11-01 --- ## Overview The input is a text prompt; the output is a 3D asset that looks like the prompt when viewed from any side. No paired text and 3D data is needed for the two variants here. Each method starts from a simple 3D representation, renders it from random cameras through a differentiable renderer, scores the renders with a pretrained model, and pushes the gradients back into the representation [arxiv:2203.13333 §3] [arxiv:2309.16585 §4.1]. The shared difficulties are the same across variants. Optimising only for the text can produce tangled geometry, because the scoring model can be satisfied by painting details onto a flat surface instead of changing the shape [arxiv:2203.13333 §1]. The other common failure is multiple faces: a second face appears on the back of a head, for example [arxiv:2309.16585 §1]. Each variant addresses these with its own constraints (see Variants). The variants differ on three axes: the representation (a mesh with texture, or a cloud of Gaussians), the scoring signal (CLIP similarity, or score distillation from a diffusion model), and any explicit 3D prior. ## Variants ### CLIP-Mesh: mesh optimisation with CLIP The representation is a Loop subdivision surface controlled by a coarse mesh, plus a texture map and a normal map. The loss is the CLIP similarity between random rendered views and the prompt, a diffusion-prior term, and a Laplacian smoothness term [arxiv:2203.13333 §3] [arxiv:2203.13333 eq. 4]. The paper adds random backgrounds, camera offsets and lighting changes so the shape must hold up from every side [arxiv:2203.13333 §3 Random Augmentations]. Card: [CLIP-Mesh](/papers/2203.13333). The authors report CLIP R-precision above Dreamfields in all four pairings of generation and evaluation CLIP models, for example 96.7 against 93.5 with ViT-B/16 on both sides [arxiv:2203.13333 tab. 1]. The evaluation uses 153 captions and a held-out view, and the prior loss is left out of those numbers [arxiv:2203.13333 §4.3]. The authors report about 50 minutes per shape on one 16 GB GPU, against more than 24 hours on four A100s for Dreamfields [arxiv:2203.13333 §4.1]. ### GSGEN: 3D Gaussians with score distillation and a 3D prior The representation is a set of anisotropic 3D Gaussians, rendered by splatting [arxiv:2309.16585 §3.2]. The scoring signal is score distillation sampling (SDS) from Stable Diffusion v1.5 [arxiv:2309.16585 §5.1]. An initial cloud from the Point-E text-to-point-cloud model sets the rough shape, and a 3D SDS term on the Gaussian centres keeps that shape during the geometry stage [arxiv:2309.16585 §4.1] [arxiv:2309.16585 §4.3]. An appearance stage then optimises colour and opacity with 2D SDS only, adding Gaussians where gaps remain and removing faint ones [arxiv:2309.16585 §4.2]. Card: [GSGEN](/papers/2309.16585). Results are visual only. The paper reports fewer multi-face artefacts and finer detail than Magic3D and Fantasia3D on the prompts it shows, and fewer multi-face failures than DreamFusion, ProlificDreamer and Fantasia3D. It reports about 40 minutes per asset on one GPU [arxiv:2309.16585 §5.2] [arxiv:2309.16585 app. A]. The multi-face problem is reduced but not removed [arxiv:2309.16585 §6]. ## When to use which | Situation | Use | Why | |---|---|---| | Output must be a mesh with a texture for a game engine or modelling tool | CLIP-Mesh | It outputs a mesh, texture map and normal map [arxiv:2203.13333 §3]. The GSGEN card describes Gaussians as the output and no mesh export [arxiv:2309.16585 §3.2]. | | No 3D prior or 3D data is available or wanted | CLIP-Mesh | Its guidance is CLIP plus a diffusion prior trained on image-text pairs [arxiv:2203.13333 §3 Diffusion Prior]. GSGEN's geometry stage relies on Point-E [arxiv:2309.16585 §4.1]. | | Coarse shape must be fixed early, with fewer multi-face failures | GSGEN | The 3D term on Gaussian centres is a direct geometric prior, and the paper reports fewer Janus artefacts on its prompts [arxiv:2309.16585 §5.2] [arxiv:2309.16585 §5.3]. | | Fine surface detail such as feathers or thatch | GSGEN | The paper shows these in qualitative comparisons with the baselines [arxiv:2309.16585 §5.2]. Its densification adds detail where SDS leaves gaps [arxiv:2309.16585 §4.2]. | | Need a published quantitative comparison | CLIP-Mesh, with caution | It reports R-precision numbers [arxiv:2203.13333 tab. 1], but only against Dreamfields. GSGEN reports no metrics [arxiv:2309.16585 §5.2]. | | Limited GPU time per asset | Either, with the author-reported numbers | CLIP-Mesh: about 50 minutes on a 16 GB P100 [arxiv:2203.13333 §4.1]. GSGEN: about 40 minutes on an RTX 3090 [arxiv:2309.16585 app. A]. The hardware and the baselines differ, so the figures are not comparable. | | Ready-to-run code with a clear licence | Check each card's code notes first | The released repositories differ from the papers in several places and need extra setup. See the code notes on [CLIP-Mesh](/papers/2203.13333) and [GSGEN](/papers/2309.16585). | ## Implementation sketch The loop is common to both variants. Only the representation, the scoring term and the schedule change between them. The sketch follows the common structure in the cards and is not a released implementation. ```pseudocode # Common optimisation loop for text-to-3D [arxiv:2203.13333 §3] # [arxiv:2309.16585 §4] def train_step(representation, prompt, step): # Render from a random camera, background and lighting [arxiv:2203.13333 §3] camera = sample_camera() image = render(representation, camera) # Score the render against the prompt. CLIP similarity for CLIP-Mesh; # SDS from a diffusion model for GSGEN [arxiv:2203.13333 eq. 1] # [arxiv:2309.16585 §4.1] loss = guidance_loss(image, prompt) # Optional 3D prior: a point-cloud or diffusion prior on the shape # GSGEN applies it to Gaussian centres only in the geometry stage # [arxiv:2309.16585 §4.1] loss = loss + prior_weight(step) * prior_loss(representation, prompt) # Shape regulariser on the mesh or the Gaussian positions [arxiv:2203.13333 eq. 2] loss = loss + regulariser_weight(step) * shape_penalty(representation) # Update and, for Gaussians, densify or prune on a schedule [arxiv:2309.16585 §4.2] update(representation, loss) if schedule_hits(step): densify_or_prune(representation) return loss ``` Where the variants differ: CLIP-Mesh has no densification step and its regulariser is a Laplacian on the surface [arxiv:2203.13333 eq. 2]. GSGEN uses a penalty on position norms and an opacity term instead, plus densification and pruning on a schedule [arxiv:2309.16585 eq. 4] [arxiv:2309.16585 §4.2]. Both papers describe their weight schedules only partly, so the schedules must be set from the stated values, not assumed. ## Contradictions - **CLIP-Mesh, elevation range.** The method section gives a range from 0° to 100° [arxiv:2203.13333 §3 Method]. The evaluation section says training is limited to 30° [arxiv:2203.13333 §4.3]. Neither is settled by the paper. - **CLIP-Mesh, prior in the ablation table.** The table row with the prior loss is headed by a B/16 generation model, but the text says the prior was not used with B/16 generation [arxiv:2203.13333 tab. 2] [arxiv:2203.13333 §4.3]. - **GSGEN, mean penalty target.** The mean penalty is described as pulling positions towards the origin in the equation and towards the geometry-stage positions in the text [arxiv:2309.16585 eq. 4] [arxiv:2309.16585 §4.2]. - **Timing claims across papers.** CLIP-Mesh reports about 50 minutes against more than 24 hours for Dreamfields [arxiv:2203.13333 §4.1]. GSGEN reports about 40 minutes, on par with Magic3D and Fantasia3D [arxiv:2309.16585 §5.2]. The baselines, hardware and prompts differ, so these numbers do not contradict each other, but they cannot be compared directly. ## Papers - [CLIP-Mesh](/papers/2203.13333): zero-shot textured mesh from text, using a Loop subdivision surface, CLIP similarity and a diffusion prior. - [Text-to-3D using Gaussian Splatting (GSGEN)](/papers/2309.16585): 3D Gaussians with SDS and a Point-E prior, in a geometry and an appearance stage.