CLIP-Mesh: Generating textured meshes from text using pretrained image-text models

In plain words

Most text-to-image systems produce pictures, not objects you can drop into a game. CLIP-Mesh tries to make the object itself. It starts from a simple ball-shaped mesh, wraps it in a random-colour texture and a flat surface-direction map, and then repeatedly changes the shape and the paint so that rendered views of the object look, to CLIP, like the text prompt. CLIP is a model that already knows how well a picture matches a sentence, so no 3D training data is needed.

Left alone, this optimisation tends to produce tangled shapes and to paint the prompt onto a flat surface instead of changing the form. The method counters this with a few tricks: it treats the ball as a smooth surface so the shape stays neat, it keeps changing the camera distance, angle, lighting and background so the shape has to hold up from every side, and it adds a second guide that proposes what a matching picture should look like.

The result is a mesh with a texture and a normal map, which is the usual format for game engines and modelling tools. The authors report that one shape takes about 50 minutes on a single 16 GB GPU, far less than the neural-field baseline they compare against.

TL;DR

• Zero-shot text-to-mesh: from a text prompt alone, it optimises a Loop subdivision control mesh, a texture map and a normal map §3.
• The loss is CLIP cosine similarity between rendered views and the prompt, plus a diffusion-prior term and a Laplacian smoothness term eq. 4.
• Randomised cameras, lighting, backgrounds and offsets are the paper's defence against texture-only solutions §3 Practical Considerations §3 Random Augmentations.
• Author-reported R-precision beats Dreamfields in all four generation/evaluation pairings in tab. 1; the prior is not used in those numbers.
• Released code is MIT, but several components it depends on are missing from the clone, and the released defaults differ from the paper in many places (see Paper vs code).

Problem

Text-to-3D with paired 3D and text data is limited by the small size of 3D datasets; the authors cite ShapeNet and CO3D as covering about 50 object categories each §1. Methods that avoid paired data are limited in other ways: CLIP-Forge stays within the 3D categories it was trained on and does not produce meshes or textures §2. Dreamfields uses a NeRF, which needs ray casting and training many network weights, and its shape and texture cannot be separated §2. The authors report more than 24 hours per shape on four A100 GPUs for Dreamfields §4.1.

Naive CLIP optimisation of a mesh gives a tangled, noisy mesh, because nothing constrains the shape and CLIP can prefer to paint small artefacts into the texture rather than deform the object §1. The authors also note a texture bias in CLIP that favours texture over shape §3 Practical Considerations.

The method assumes a differentiable renderer and differentiable CLIP encoders, so gradients reach the vertices and texture §3.

Method

The scene is a triangle mesh with three parts: control vertices V0 that define a Loop subdivision limit surface V = S(V0), a texture map T, and a normal map. The limit surface is differentiable and smooth, which acts as an implicit regulariser against triangle inversion §3.

1. Render. Random cameras (azimuth, elevation, distance, field of view) view the limit surface through a differentiable renderer R, producing images I §3. The rendering equation is unnumbered in the paper.
2. Encode. The CLIP image encoder maps each image to E and the CLIP text encoder maps the prompt to e_t. Both encoders end in normalisation, so the scores are cosine similarities §3.
3. Text loss. The average negative cosine similarity between the view embeddings and the prompt embedding eq. 1.
4. Smoothness. A uniform Laplacian over one-ring neighbours, averaged over vertices eq. 2 §3 Laplacian Regularizer. The weight lambda_t decays from an initial value towards a minimum, following lambda_t = (lambda_{t-1} − lambda_min)·10^(−kt) + lambda_min §3 Laplacian Regularizer. The paper says the initial value and decay parameters are tunable; k is not stated.
5. Prior loss. A diffusion prior, trained on a 400 million image-text pair dataset, samples a CLIP image embedding ê_k conditioned on e_t. The loss is the same negative cosine similarity against ê_k eq. 3 §3 Diffusion Prior. The prior follows the configuration of the cited hierarchical text-to-image work, scaled down; its size is not stated §3 Practical Considerations.
6. Total objective. The CLIP loss, the Laplacian term weighted by lambda_t, and the prior loss weighted by alpha eq. 4. The paper gives no value for alpha.
7. Augment. Each view draws a random background (Gaussian noise, a solid colour, or a random colour checkerboard), a random position offset, a random field of view, and a random distance. Rendering is at 512 px and the image is then downscaled to 224 px for CLIP §3 Random Augmentations §3 Practical Considerations.
8. Multiple objects. Each mesh in a scene can have its vertices, texture and normal map frozen or optimised separately §4.2.

L_{CLIP}(\bm{V},\bm{T},\bm{\tilde{T}},\bm{p}) = -\frac{1}{K}\sum_{\bm{e}_i \in \bm{E}}{\bm{e}_i^T \bm{e}_t} eq. 1

L_{\delta} = \frac{1}{N} \sum_{i=1}^{N} \Vert \delta_i \Vert^2 eq. 2

L_{PRIOR}(\bm{V},\bm{T},\bm{\tilde{T}},\bm{p}) = -\frac{1}{K}\sum_{\bm{e}_i \in E}{\bm{e}_i^T \bm{\hat{e}}_k} eq. 3

\min_{\bm{V}_0,\bm{T},\bm{\tilde{T}}} L_{CLIP}(S(\bm{V}_0),\bm{T},\bm{\tilde{T}},\bm{p}) + \lambda_t L_{\delta}(\bm{V}) + \alpha L_{PRIOR}(S(\bm{V}_0),\bm{T},\bm{\tilde{T}},\bm{p}) eq. 4

Here delta_i = v_i − (1/|N_i|) Σ_{j in N_i} v_j, N_i is the one-ring of v_i and N is the vertex count §3 Laplacian Regularizer. The paper defines K nowhere; see Pitfalls.

The optimisation loop, one step. The paper gives no algorithm box, so this is reconstructed from §3 and the code.

# One optimisation step  [§3] [code: loop.py#L232-L539]
# Input:  control vertices V0 (n, 3); texture T (512, 512, 4); normal N
# Output: updated V0, T, N

def train_step(V0, T, N, text_embed, prior_embed):
    # Limit surface of the Loop subdivision of V0; differentiable  [§3]
    surface = limit_surface(V0)                          # (m, 3)
    # Draw 25 random cameras, backgrounds and light positions  [§3]
    views = sample_views(batch_size=25)
    # Render each view at 512 px  [§3] [code: loop.py#L386-L441]
    images = render(surface, T, N, views)                # (25, 512, 512, 3)
    # Downscale to the CLIP input size  [§3] [code: loop.py#L443-L473]
    images = resize(images, size=224)                    # (25, 224, 224, 3)
    # Normalised CLIP image embeddings  [§3]
    image_embeds = normalise(clip_image(images))         # (25, 512)
    # Mean negative cosine similarity to text and prior  [eq. 1] [eq. 3]
    clip_loss = -mean(image_embeds @ text_embed)
    prior_loss = -mean(image_embeds @ prior_embed)
    # Uniform Laplacian; the paper applies it to the limit surface, the
    # code to V0 (see Paper vs code)  [eq. 2] [code: loop.py#L317, L512]
    laplacian = mean(norm(delta(V0)) ** 2)
    # Total objective: CLIP weight 1, prior weight alpha  [eq. 4]
    loss = clip_loss + alpha * prior_loss + lambda_t * laplacian
    # Adam update on vertices and texture maps  [§3]
    V0, T, N = adam_update(loss, V0, T, N)
    # Clamp texture to [0, 1], normal to [-1, 1]  [code: loop.py#L535-L537]
    T = clip(T, 0, 1)
    N = clip(N, -1, 1)
    return V0, T, N

The Laplacian weight schedule, one update. The code applies it per step, and the paper per epoch.

# Laplacian weight decay  [§3 Laplacian Regularizer] [code: loop.py#L515-L519]
# Input:  previous weight, step index t, minimum weight
# Output: weight for this step

def next_laplacian_weight(weight_prev, step, weight_min):
    # Geometric decay towards the minimum; the code uses factor 1e-6 per step
    decay = 10 ** (-step * 1e-6)                         # paper's k not stated
    return (weight_prev - weight_min) * decay + weight_min

Hyperparameters

Values marked "code" come from the released configuration configs/paper.yml, which its own header says was used for the paper's figures paper.yml. Where the paper does not give a value, the table says so.

Results

Author-reported. Setup: CLIP R-precision on 153 captions taken from Dreamfields, two shapes per caption, a held-out elevation of 45°, and no prior loss in these numbers §4.3. Higher is better. Table entries are the paper's; the Dreamfields row is the paper's comparison row, and its source of numbers is not stated tab. 1.

Ablation, one shape per caption §4.4. Column headers are evaluation models; the table is headed "Method (CLIP B/16)" tab. 2.

Other reported numbers:

• Speed: each shape took about 50 minutes on one 16 GB P100 GPU; Dreamfields took over 24 hours on four A100 GPUs. The authors put the speed-up at about 100× for similar configurations §4.1.
• Figures: results are shown for single objects, multiple objects in one scene, and comparisons with Dreamfields, one with their published images and one with new prompts run through Dreamfields' released code at its medium configuration §4.1 fig. 1 fig. 3 fig. 4 fig. 5 fig. 6 fig. 7 fig. 8.
• The authors state that the prior loss is left out of the Table 1 setting because the prior was trained on ViT-B/32 embeddings only §4.3.

Paper vs code

Compared against the official repository at commit e588c12d3c24ce21d9d3452f1e5b83c892be92f4 (MIT).

• Learning rate. Paper starts at 0.001 §3 Practical Considerations; the config's default is 0.01 and the schedule multiplies by 10^(−0.0002) per step, so the rate is 0.1× of the start after 5000 steps paper.yml loop.py.
• Camera distance. Paper 3.0 to 7.0 §3 Practical Considerations; code 5.0 to 12.0 paper.yml.
• Field of view. Paper 30° to 60° §3 Practical Considerations; code 30° to 90° paper.yml.
• Elevation. Paper's range is 0° to 100° §3 Method; code caps at 90° paper.yml. §4.3 says training is limited to 30° §4.3, which matches neither.
• Prior sampling time. Paper says the prior is sampled "during optimization time" §3 Diffusion Prior. Code samples once, before the loop, and reuses that single embedding for every step and every view loop.py loop.py.
• Prior sampling procedure. Paper says the prior is run "for N timesteps" from noise §3 Diffusion Prior; code calls a full reverse sampling loop over the configured 1000 timesteps loop.py paper.yml.
• Laplacian target. Paper's eq. 2 and eq. 4 apply it to the limit surface V eq. 2 eq. 4. Code builds the regulariser from the control mesh m, which holds V0 loop.py loop.py. The regulariser's implementation is in the nvdiffmodeling submodule, which is missing from the clone (see Pitfalls), so the code's exact operator cannot be confirmed here.
• Laplacian schedule. Paper's decay is per epoch with an unstated k §3 Laplacian Regularizer; code decays per step with 10^(−1e-6) loop.py.
• Texture blur. Not in the paper. Code applies a Gaussian blur to every texture map at every step, before rendering loop.py paper.yml.
• Shape-only warm-up. Not in the paper's main text. For the first half of the steps, half the batch is rendered with a flat 0.5-grey diffuse texture and one peeling layer loop.py loop.py loop.py loop.py paper.yml.
• Alpha initialisation. The config has init_c: 0.85, described as the initial alpha value paper.yml. The loop does not read it; the texture starts uniform random on all channels loop.py. The paper gives no alpha value.
• Sphere vertex count. Paper says 600 §3 Practical Considerations; the shipped sphere has 642 vertices sphere.obj.
• Up vector. Paper says the y-axis §3 Practical Considerations; code uses (0, −1, 0), i.e. negative y camera.py.
• Position offset. Paper gives no range §3 Random Augmentations; code uses ±floor(dist_min/2), which is ±2 world units with the default dist_min of 5 camera.py paper.yml.
• Prior size. Paper says the prior is scaled down §3 Practical Considerations; code fixes a 512-dimension, 12-layer network paper.yml. The released weight file is named larger-model.pth in the README README.md. The paper does not say which weights it used.
• Loss terms. The CLIP loss is the batch mean of negative cosine similarity loop.py helpers.py, which matches eq. 1 with K as the batch size. The prior term is the same form loop.py. The total is clip + 0.33 × prior + lambda_t × Laplacian loop.py.
• Evaluation code. The repository has no R-precision script, and the 153 captions are not included [repo-wide search for "precision", "retriev", "R-Prec", "evaluat"].
• Dependencies. The pinned OpenAI CLIP and nvdiffrast commits are in requirements.txt requirements.txt. The README says Python 3.7 and PyTorch 1.11 README.md.

The loss forms of eq. 1 and eq. 3, and the total of eq. 4, match the code (see the Loss terms bullet). The CLIP normalisation constants are as used by CLIP loop.py loop.py.

Pitfalls

• K is undefined. Eq. 1 and eq. 3 divide by K without saying what it is. The code uses the batch size loop.py.
• Prior is one sample per run in the code. Repeating the paper's "during optimization" literally would sample a new embedding each step. The code does not; this changes the prior's role from a stochastic signal to a fixed target loop.py.
• The prior is tied to ViT-B/32 embeddings. The paper says the prior was trained only on B/32 embeddings and was therefore left out of the B/16 generation setting §4.3. The code hard-codes a 512-dimension embedding and does not check that clip_model matches the prior loop.py loop.py. A mismatched model with the same embedding size runs without an error.
• Table 2 conflicts with §4.3. The "Prior loss" row is headed "Method (CLIP B/16)", yet §4.3 says the prior was not used with B/16 generation. The B/16 column also drops from 92.1 to 91.5 at the prior row, while the text says the prior gives the best overall results tab. 2 §4.4.
• Two elevation ranges. §3 gives Beta over 0° to 100°, §4.3 says training is limited to 30° §3 Method §4.3. The code default is 90° paper.yml.
• Laplacian target. Paper says V; code says V0 (see Paper vs code). An implementer following the paper will get a different regulariser.
• Missing submodules. The clone of the official repository has empty submodules for loop_limitation (limit surface), nvdiffmodeling (renderer, mesh, regulariser) and DALLE2-pytorch (prior) .gitmodules. The limit-surface and Laplacian operators therefore cannot be checked against the released code. These submodules are not cited here.
• Prior weights come from outside the repo. The README downloads them from a Hugging Face space README.md; their licence and training provenance are not checked here. The prior training code is not in this repository.
• Non-determinism. The code sets seeds and cuDNN determinism main.py, but renderer and CUDA kernels can still vary. The paper gives no seed.
• Memory. The FAQ says to reduce train_res, batch_size or texture_resolution on limited memory FAQ.md. The paper does not discuss memory.
• Non-manifold meshes and odd valences break limit subdivision. Vertex valence must be in roughly 3 to 50, and the mesh must be manifold FAQ.md. The paper does not state this constraint.
• Figure and table numbers. Only the LaTeX source was fetched. Figure and table numbers are counted in source order (teaser as Figure 1); they were not checked against the compiled PDF.
• Quantitative results depend on an unreleased evaluation. Table 1 uses R-precision on 153 captions and a held-out 45° view; neither the captions nor the evaluation script is in the repository.
• Genus is fixed by the template mesh. The authors note the genus is set by the initial sphere, and that a transparency channel is a partial workaround §5 Genus.
• Artefacts from the image model. Text can appear on the object, and the pyramid in the figures has small people on its side §5 CLIP Limitations.
• Cost. The authors say about 50 minutes per shape on a 16 GB P100, against more than 24 hours for Dreamfields on four A100s §4.1. The number of steps is not stated in the paper; the code default is 5000 paper.yml.

Lineage

• Builds on: CLIP §2 (a joint image-text encoder, cited as the basis of the scoring); Loop subdivision §3 and its limit-surface evaluation §3; Laplacian regularisation following the cited prior work §3 Laplacian Regularizer; a diffusion prior following the cited hierarchical text-to-image work §3 Diffusion Prior. None of these has a card in this wiki, so nothing from them is used here.
• Compared against: Dreamfields (Jain et al., cited in §2 and §4), which is a NeRF-based zero-shot text-to-3D method. Its arXiv id is not given in the source.
• Superseded by: not recorded in this wiki.