Text2Mesh: Text-Driven Neural Stylization for Meshes

In plain words

Text2Mesh changes how an existing 3D shape looks without changing its basic form. You give it a mesh, such as a vase, and a sentence, such as "a vase made of wicker". It then adds color and small surface bumps that match the sentence.

A small neural network decides, for each point on the surface, which color it should have and how far to push the surface in or out. To judge a result, the method renders the shape from a few nearby camera angles and asks CLIP, a model that scores how well an image matches a text, whether the renders fit the sentence. Training pushes that score up. The shape itself never changes, so the output keeps the original structure while taking on the new look.

TL;DR

Text2Mesh stylizes an existing mesh from a text prompt. It trains an MLP that maps each vertex to an RGB color and a bounded offset along the normal. Gradients come from CLIP similarity between the prompt and augmented multi-view renders. No pretrained generator or 3D dataset is needed. Official MIT-licensed code exists, but its defaults differ from the paper in several places, so the demo scripts are the safer reference (see Paper vs code).

Problem

Text-guided editing of a 3D mesh's color and fine geometry, without a 3D generative model or a dataset of stylized meshes §1 §3. The authors observe that plain CLIP optimization on a mesh converges to degenerate, noisy solutions: the joint embedding has many false positives, where noise scores as a good match §1. Augmentations and a geometric prior are what keep the result meaningful §4.2. The paper also says that no prior method performs this task, so it compares against a VQGAN-CLIP adaptation §4.3.

The approach assumes the source geometry and the target style are compatible. For a mismatched prompt, the stylization can ignore the shape and erase its content; adding the object category to the prompt (for example "stained glass dragon") preserves content §4.6.

Method

Content is the input mesh, with vertices V and faces F, which stays fixed. Style is a neural style field (NSF) that maps every surface point to a color c_p in [0, 1]^3 and a displacement d_p along the vertex normal §3.1. The stylized mesh is rendered with an interpolation-based differentiable renderer, and the renders are scored with CLIP §3.1 §3.2 fig. 4.

Steps:

1. Normalize the input into a unit box §3.1. The code uses a bounding sphere instead (see Paper vs code).
2. Encode each vertex with Fourier features eq. 1 §3.1. The frequency scale is a hyperparameter, which the user controls §4.1 fig. 7.
3. Shared trunk N_s feeds two branches: N_c for color and N_d for displacement §3.1 app. D.1.
4. Apply the style. Color is added to a grey base; each vertex moves by d_p times its normal. The displacement is limited to (-0.1, 0.1) §3.1 app. D.1.
5. Build a displacement-only mesh that keeps the geometry but is rendered grey, so its loss can only change the geometry §3.1 §3.2.
6. Sample views. Find an anchor view: the paper renders uniform views around a sphere and takes the one with the highest CLIP score §3.3. Then sample n_θ = 5 views from a Gaussian around the anchor, with σ = π/4 §3.3.
7. Render two images per view: the colored mesh and the grey displaced mesh §3.2.
8. Augment with ψ_global (a random perspective transform) and ψ_local (a random perspective transform plus a random crop to 10% of the image) §3.3 app. D.2.
9. Embed with CLIP and average over views: Ŝ^full, Ŝ^local and Ŝ^displ eq. 2 eq. 3 eq. 4 §3.2. The target is φ_target = E(t) §3.2.
10. Loss: the sum of cosine similarities between the three embeddings and the target eq. 5 §3.2. The Ŝ^displ term updates only N_s and N_d; the other two update N_s, N_c and N_d §3.2. The paper repeats the augmentation step n_aug times per iteration, but gives no value §3.2.
11. Optimize with Adam for 1500 iterations app. D.2.

Equations (symbols from the paper; the view symbols are written here in our notation: I^full_θ is the colored render, I^displ_θ the grey render, for view θ):

\gamma(p) = \left[\cos(2\pi \mathbf{B}p),\ \sin(2\pi \mathbf{B}p)\right]^{\top} eq. 1

where B has entries drawn from N(0, σ²). The paper gives the shape of B as n × 3 but does not define n in the text.

\hat{S}^{\text{full}} = \frac{1}{n_\theta} \sum_{\theta} E\left(\psi_{\text{global}}(I^{\text{full}}_\theta)\right) eq. 2

\hat{S}^{\text{local}} = \frac{1}{n_\theta} \sum_{\theta} E\left(\psi_{\text{local}}(I^{\text{full}}_\theta)\right) eq. 3

\hat{S}^{\text{displ}} = \frac{1}{n_\theta} \sum_{\theta} E\left(\psi_{\text{local}}(I^{\text{displ}}_\theta)\right) eq. 4

\mathcal{L}_{\text{sim}} = \sum_{\hat{S}} \text{sim}\left(\hat{S}, \phi_{\text{target}}\right) eq. 5

sim is cosine similarity §3.2. E is the CLIP image encoder, with 512-dimensional embeddings §3.2. The paper states the loss as a sum of similarities to maximize, so an implementation minimizes its negative.

Pseudocode (sketch; the paper has no algorithm box):

# Text2Mesh training loop (sketch, not an algorithm box in the paper)
# Input:  vertices (N, 3), faces, target text t
# Output: vertex colors (N, 3), displaced vertices (N, 3)

def stylize(vertices, faces, target_text, n_iter):
    # Normalize to a unit box; the code uses a sphere  [§3.1]
    vertices = normalize_to_unit_box(vertices)
    # Target embedding from CLIP  [§3.2] [eq. 5]
    target = clip_encode_text(target_text)       # (512,)
    # Fourier encoding, shared trunk and two branches  [§3.1] [eq. 1]
    field = StyleField()
    # Adam, learning rate 5e-4  [app. D.2]
    optimizer = Adam(field.parameters(), lr=5e-4)
    for step in range(n_iter):                   # 1500  [app. D.2]
        # Per-vertex color and offset; offset bounded to (-0.1, 0.1)  [§3.1]
        colors, offsets = field(vertices)        # (N, 3), (N, 1)
        # Move each vertex along its normal  [§3.1]
        displaced = vertices + offsets * normals # (N, 3)
        # Five views around the anchor, Gaussian spread pi/4  [§3.3]
        views = sample_views(anchor, count=5, sigma=pi / 4)
        # Colored render and grey displaced render per view  [§3.2]
        full = render(displaced, colors, views)
        grey = render(displaced, gray, views)
        # Global augmentation on the full view, averaged  [§3.2] [eq. 2]
        s_full = mean(clip(perspective(full)))
        # Local augmentation (10% crop) on the full view  [§3.3] [eq. 3]
        s_local = mean(clip(crop_perspective(full)))
        # Local augmentation on the grey view; geometry only  [§3.2] [eq. 4]
        s_displ = mean(clip(crop_perspective(grey)))
        # Maximize similarity: minimize its negative  [eq. 5]
        loss = -sum(cosine(s, target)
                    for s in [s_full, s_local, s_displ])
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    return colors, displaced

Hyperparameters

Paper values come first; code values follow where they differ. "Not stated" means the paper and the released code do not give a value.

Results

Author-reported. Setup: 57 users rated 8 random pairs of source mesh and style prompt. Scores are 1 to 5 on three questions; control pairs (mismatched image and text) scored 1.16 §4.3 tab. 1. The baseline is VQGAN-CLIP extended to meshes: it colors inside a binary 2D mask projected from the source shape and does not deform it §4.3.

Differences are 1.07, 0.44 and 1.32 for Q1 to Q3 §4.3.

Ablation CLIP similarity (the paper's own score, against the target) on one candle example, values as annotated in fig. 5: full 0.36; without the style network 0.26; without 2D augmentations 0.20; without Fourier features 0.26; without local crops 0.30; without the geometry-only term 0.29; without 3D geometry prior 0.29. The authors note that the no-3D variant still scores high, which shows CLIP can reward degenerate solutions §4.2.

Interplay of geometry and color, one alien example, values as annotated in fig. 11: full 0.322, geometry-only 0.250, color-only 0.320 §4.2.

Cost, from the paper: about 25 minutes on one RTX 2080 Ti for 1500 iterations app. D.2; under 25 minutes overall on one GPU, with high-quality results often within 10 minutes §4. The paper's test meshes average 79,366 faces §4.

Paper vs code

Code: github.com/threedle/text2mesh at code_commit, MIT. The repository is linked from the project page that the abstract names, and its owner is the authors' lab.

• Normalization. Paper: unit bounding box §3.1. Code: bounding sphere, centered on the mean and scaled by the largest distance from it Normalizer.py. The code normalizes the input main.py and renormalizes each displaced mesh every iteration main.py.
• Output frame. The exported vertices stay in the normalized frame. The code defines an inverse normalizer, which is never called Normalizer.py main.py.
• Positional encoding. Paper: B entries from N(0, σ²), feature 256-dim §3.1 app. D.1. Code: B is drawn with standard deviation sigma, its columns are sorted by norm (order only), and the output concatenates the raw input with sin and cos utils.py utils.py.
• Progressive encoding. Not in the paper. Code enables it by default, which multiplies each Fourier frequency by a ramp that goes from 0 to 1 as training proceeds main.py neural_style_field.py neural_style_field.py. The demos for person meshes turn it off with --no_pe run_all.sh.
• Trunk depth. Paper: 4 shared layers app. D.1. Code: 5 linear layers in the trunk at default depth neural_style_field.py.
• View averaging. Paper averages CLIP embeddings over views eq. 2 §3.2. Code's default averages per-view cosine similarities instead. Embedding averaging needs --clipavg view, which all demo scripts pass main.py run_shoe.sh.
• Anchor view. Paper selects the anchor by CLIP score over uniform views §3.3 app. C. Code has no selection step; the anchor is a manual argument main.py and the demos pass one per run run_shoe.sh.
• View spread. Paper gives one σ = π/4 §3.3. Code perturbs elevation with σ = π/std and azimuth with σ = 2π/std, with the first view unperturbed; the demos use std 4 render.py run_shoe.sh.
• Loss weighting. Paper's Ŝ terms have equal weight eq. 5 §3.2. In code, the global term uses one augmentation, while the local and displacement terms sum four augmented similarities each, so they weigh about four times as much in the demos main.py main.py main.py.
• Displacement term. Paper always includes it §3.2. Code enables it only with --geoloss, which is off by default main.py main.py; all demos pass it run_shoe.sh.
• Learning rate decay and iteration count. Paper: decay 0.9 every 100 iterations, 1500 iterations app. D.2. Code defaults: no decay and 6000 iterations main.py main.py. The demos match the paper run_shoe.sh.
• Logged loss. The saved loss curve holds only the global term, not the local or displacement terms main.py main.py.
• Crop. Paper: "10% of the original image" §3.3. Code uses torchvision's RandomResizedCrop with scale 0.1 main.py main.py. torchvision's documented meaning of scale is a fraction of area, so the side is about 32% of the image.
• Lighting. Paper does not mention shading. Code shades every render with spherical-harmonic lighting render.py.
• Modalities. Paper describes image and mesh targets §4.4 fig. 12 fig. 13. Code has --prompt and --image only main.py; no mesh-target path exists.
• Unused flag. --frontview is parsed but never read main.py.
• Symmetry. Matches §4.5, applied to z only main.py.
• Standardization. Code has a z-score option that the paper does not describe; the person demos use it main.py run_all.sh.

The three embedding terms are implemented as described, the geometry-only term does not reach the color branch, and the color and displacement output ranges match. The discrepancies above concern defaults, weighting, normalization and the view and anchor setup.

Pitfalls

• Default flags are not the paper's setup. Running main.py with defaults gives 6000 iterations, no learning-rate decay, mean-of-cosines averaging, and no geometry term. Use the demo flags for a paper-like run run_shoe.sh.
• Unstated augmentation count. The paper gives no value for n_aug §3.2. The demos use 1 global and 4 local/displacement augmentations, summed. Changing this rebalances the terms.
• Perspective parameters. The paper does not give the perspective strength or fill. Code uses probability 0.8, distortion 0.5 and white fill main.py. The white fill may matter for the displacement-only view.
• Progressive encoding ramp. My reading of the code (not run): with width 256 and 1500 iterations, the ramp reaches full strength near iteration 87 neural_style_field.py neural_style_field.py. Later training runs with all frequencies active. Confirm before relying on it.
• Geometry-only term is essential. The paper's ablation shows that without it, the network simulates geometry with shading §4.2 fig. 5. Keep --geoloss on.
• Augmentation is essential. Without 2D augmentations, the result follows no target text §4.2 fig. 5.
• CLIP scores are weak evidence. The paper shows a degenerate solution scoring high §4.2. Judge outputs visually.
• Triangle size sets the resolution. Large triangles give low-quality stylization. The README recommends remeshing first README.md.
• Normalization changes the scale of the bumps. Displacement is bounded in normalized units, so the physical size of a bump depends on the normalization, which the paper and the code define differently (see Paper vs code). The exported vertices are normalized, not in the input's units main.py.
• Environment. Installation needs a CUDA GPU; the README states the install fails elsewhere, and lists Python 3.7 and CUDA 11 README.md.
• Societal bias. The authors observed gender bias in some results, for example a "nurse" style that adds female features to a male input app. F fig. 25. Stylizations inherit CLIP's biases.
• Mesh quality. The paper handles non-manifold meshes with boundaries and arbitrary genus, and needs no UV parameterization §1 §3.1.
• Results are author-reported. The user study data and the VQGAN baseline code are not in the repository.

Lineage

• Builds on: CLIP, as the text-image similarity used in the loss §1 §3.2; Fourier feature mapping for the spectral bias of MLPs §3.1; the neural prior idea that an MLP favours smooth solutions §1.
• Compared against: a VQGAN-CLIP adaptation to meshes, used as the only baseline §4.3.