NeuMesh: Learning Disentangled Neural Mesh-based Implicit Field for Geometry and Texture Editing

In plain words

NeuMesh represents an object as a neural field that is tied to the triangle mesh of its surface. Instead of one large network that describes every point in space, each mesh vertex stores a small set of learned numbers: one set for shape and one for appearance. To render a point near the surface, the method blends the numbers of the nearest vertices and passes them to two small networks. One estimates how far the point is from the surface, and the other gives its colour.

The point of this design is that the mesh stays an ordinary, editable object. Moving vertices in standard 3D software moves the rendered object with them, and no retraining is needed for that. The same structure lets you copy the look of one region onto another, fill a region with a sample texture, or paint on a photo and carry the paint into the 3D object.

Training first copies the shape from a neural surface model that was trained earlier, and then fine-tunes the mesh-based model on the photos. Training directly from scratch was reported to produce artifacts.

TL;DR

NeuMesh is a neural implicit field whose geometry and texture codes sit on the vertices of a prior triangle mesh, so parts of an object can be edited by moving the mesh or re-assigning codes. It is trained by distilling a NeuS teacher and then fine-tuning on photos. Editing covers mesh-guided geometry changes, texture swapping, filling and painting from a single view. Author-reported results beat NeuS and NeuTex on DTU in PSNR, SSIM and LPIPS, and beat NeuS on PSNR and LPIPS (not SSIM) on NeRF-synthetic. Mesh Chamfer distance is close to NeuS. The official code is MIT-licensed, and several of its training settings differ from the paper, see Paper vs code.

Problem

Neural rendering methods that allow editing usually support only rigid transformations, work only for simple shapes, or do not support fine-grained texture editing §2. Methods that rely on UV mapping are limited on complex topology and delicate structure §2. Methods that use an MVS mesh only as a geometry proxy for image features cannot support high-level editing §2.

The approach assumes a usable prior triangle mesh of the target object. It is produced from a neural SDF trained with NeuS and extracted with marching cubes §3.1 app. A. Training with the mesh-attached representation from scratch leads to artifacts and sub-optimal convergence §3.2.

Method

NeuMesh encodes a neural implicit field on a mesh scaffold. Geometry and colour are kept in two separate latent spaces, decoded by two MLPs §3.

Representation §3.1

1. Build the scaffold. Train NeuS on the multi-view images, extract a mesh with marching cubes at 256 voxels per side over the range [-1, 1] app. A, and obtain a mesh of about 50K to 150K vertices §3.1.
2. Store per vertex v_k: a geometry code (32 dimensions), a texture code (32 dimensions) and a sign vector n_k that starts at the vertex normal §3.1 app. A.
3. For a query point x, find its K = 8 nearest vertices app. A and weight them by inverse distance, normalised over the neighbours eq. 1.
4. Interpolate the two codes with the same weights eq. 1.
5. Compute an interpolated signed distance h̃(x) eq. 3. Each neighbour k forms the offset p_k = x - v_k and blends its sign vector with that offset, using weights ω^n = 0.1 and ω^p = ‖p_k‖ §3.1. The neighbour value h_k is the dot product of p_k with the blend. h̃ is the inverse-distance-weighted sum of the h_k eq. 3.
6. Decode the SDF s = F_G(geometry code, h̃) §3.1. Decode the colour c = F_R(texture code, h̃, view direction d, ∇s) §3.1.
7. Apply positional encoding to the interpolated codes, the distance and the direction before the MLPs §3.1 app. A.
8. Render each ray with an opacity derived from the SDF: α_j = max((Φ_s(s_i) − Φ_s(s_{i+1})) / Φ_s(s_i), 0) and transmittance T_i = ∏_{j<i} (1 − α_j). The colour is the sum of T_i α_i c_i eq. 2. Φ_s is the logistic CDF, and s is the sharpness (see Pitfalls) §3.1.

Training §3.2

9. Train a NeuS teacher per object. Supervise NeuMesh with the teacher's SDF and colour at every sample point (distillation loss L_d), and with the ground-truth pixel colours (photometric loss L_f) eq. 4 §3.2 app. B.1.
10. Regularise the sign vectors towards the pre-computed vertex normals (L_rs), and add an Eikonal term that pushes the SDF gradient norm to 1 (L_re) §3.2.
11. Minimise the weighted sum of the four terms eq. 5 §3.2. For DTU, a binary cross-entropy term on foreground masks is also added app. B.1.

Mesh-guided geometry editing §3.3 app. B.2

12. Deform the mesh with an as-rigid-as-possible method (ARAP) or with Blender. No fine-tuning is needed for the field to follow §3.3.
13. For each moved vertex, compute the change of its averaged normal and add that change to the interpolated signed distance h̃, so local distances stay consistent app. B.2.

Designatable texture editing §3.4 app. B.3

The edit replaces the texture codes l^t in the selected region and may also replace the radiance decoder F_R there §3.4.

14. Swapping. Mark the source and target regions on the mesh. Annotate 4 to 9 point correspondences, align with Umeyama's scaled rigid method and then ARAP, and give each target vertex the inverse-distance blend of the codes of its 4 nearest deformed source vertices app. B.3.
15. Filling. Unwrap a UV map for the selected vertices in Blender. Tile a small template patch of about 10 vertices from a pretrained model over the UV map in a sliding window. Bind the radiance decoder of the pretrained model, because the template and target codes do not share a latent radiance space. Blend colour from two decoders by inverse distance near the boundary app. B.3.
16. Painting. Shoot rays through the painted pixels, collect the vertices of the hit faces, and fine-tune only those codes with Adam at a fixed learning rate of 0.01 for 8000 iterations, about one hour app. B.3. Training rays are restricted to a slightly dilated paint mask, and random view directions are fed to F_R §3.4.

The core sampling is the same for rendering, editing and training. The pseudocode below follows the steps above.

# Query one point x (N, 3) and its view direction  [§3.1] [eq. 1] [eq. 3]
# Stored per vertex: geo_code (V, 32), tex_code (V, 32), sign (V, 3)
def query_point(x, view_dir, mesh):
    # 1. K = 8 nearest vertices  [app. A]
    idx = nearest_vertices(x, mesh.vertices, k=8)      # (N, 8)
    # 2. Inverse-distance weights, normalised over the neighbours  [eq. 1]
    w = 1.0 / distance(x, mesh.vertices[idx])          # (N, 8)
    w = w / w.sum(dim=-1, keepdim=True)
    # 3. Interpolate the geometry and texture codes  [eq. 1]
    geo = (w[..., None] * geo_code[idx]).sum(dim=-2)   # (N, 32)
    tex = (w[..., None] * tex_code[idx]).sum(dim=-2)   # (N, 32)
    # 4. Offset to each neighbour, and its blend with the sign vector  [eq. 3]
    offset = x[:, None] - mesh.vertices[idx]           # p_k, (N, 8, 3)
    w_p = norm(offset)                                 # omega^p, (N, 8, 1)
    blend = (0.1 * sign[idx] + w_p * offset) / (0.1 + w_p)
    h_k = (offset * blend).sum(dim=-1)                 # (N, 8)
    # 5. Interpolated signed distance  [eq. 3]
    h = (w * h_k).sum(dim=-1, keepdim=True)            # (N, 1)
    # 6. Decoders: SDF from geometry, colour from texture  [§3.1]
    sdf = F_G(encode(h), encode(geo))
    colour = F_R(encode(tex), encode(h), encode(view_dir), grad(sdf, x))
    return sdf, colour

# Render one ray  [§3.1] [eq. 2] [app. A]
def render_ray(origin, direction, sharpness):
    # 1. 64 coarse samples between the near and far bounds  [app. A]
    # 2. 64 fine samples added by progressive up-sampling  [app. A]
    points = sample_points(origin, direction)          # (M, 3), M = 128
    sdf, colour = query_point(points, direction, mesh)
    # 3. Opacity from the logistic CDF of the SDF  [eq. 2]
    cdf = sigmoid(sharpness * sdf)                     # Phi_s(s_i)
    alpha = max((cdf[:-1] - cdf[1:]) / cdf[:-1], 0)    # (M - 1,)
    # 4. Exclusive transmittance and colour sum  [eq. 2]
    transmittance = exclusive_cumprod(1 - alpha)       # T_i
    return sum(transmittance * alpha * colour[:-1])

# One training step  [§3.2] [eq. 4] [eq. 5] [app. B.1]
def training_step(rays, targets, teacher):
    # 512 rays per step  [app. B.1]
    colours, samples = render_rays(rays)
    # Distillation: match the teacher's SDF and colour at each sample  [eq. 4]
    teacher_sdf, teacher_col = teacher(samples.xyz, samples.dirs)
    loss_distill = (
        sum(norm(samples.sdf - teacher_sdf))
        + sum(norm(samples.colour - teacher_col))
    )
    # Photometric loss against the ground-truth pixels  [eq. 4]
    loss_photo = sum(squared_norm(colours - targets))
    # Sign vectors stay close to the initial vertex normals  [§3.2]
    loss_sign = sum(squared_norm(sign - normal_init))
    # Eikonal term: the SDF gradient should have unit norm  [§3.2]
    loss_eikonal = sum(squared(norm(grad_s) - 1.0))
    # Weights 1, 1, 0.01, 0.01  [eq. 5] [§3.2]
    return (
        1.0 * loss_distill + 1.0 * loss_photo
        + 0.01 * loss_sign + 0.01 * loss_eikonal
    )

Hyperparameters

Results

Author-reported. Setup: DTU (15 scenes, 1600 × 1200 images, foreground masks, 10% of images held out for testing §4.1) and NeRF 360° Synthetic (four scenes: Mic, Lego, Chair, Hotdog §4.1). Baselines are NeuS and NeuTex §4.2. Mesh quality is compared with NeuS and NeuTex, and NeuS was retrained on the training split, so its numbers differ from the NeuS paper tab. 2.

Rendering quality tab. 1. The table does not say which scenes form the NeRF-synthetic average.

Mesh quality, Chamfer distance per DTU scan tab. 2

NeuS* was retrained on the training split tab. 2.

Ablation, two scenes tab. 3

Design alternatives app. D

Other claims: geometry editing keeps the field aligned with the moved mesh without fine-tuning §3.3 app. B.2. Texture-code swapping was compared with warping the radiance field, which showed visible artifacts app. D. On DTU Scan 114, reducing the mesh to 10K vertices slightly lowers the rendering metrics, yet the authors report it still beats NeuS there app. D. Limitations: no lighting or specular modelling, and objects that the reconstruction fails on (smoke, liquid) cannot be represented §5.

Paper vs code

Repository: https://github.com/zju3dv/neumesh, linked from the authors' project page, at commit 579c545e4abf4666ec5fe784ad87c0454adc15d3. Its LICENSE is MIT (copyright 2022 Bangbang Yang, Chong Bao) LICENSE. I checked the training loss, sampling, distance and scheduler code against the paper.

• Photometric loss. The paper writes a squared L2 loss eq. 4. The code uses L1 on pixel colours trainer.py.
• Distillation. The paper writes sums of norms over rays and samples eq. 4. The code uses L1 on the SDF trainer.py and mean squared error on colour trainer.py, both as means.
• Eikonal weight. The paper gives λ_re = 0.01 §3.2. The DTU config sets 0.1 neumesh_dtu_scan63.yaml.
• Sign regularisation. The paper gives λ_rs = 0.01 with a sum of squared norms §3.2. The DTU config sets 0.001 neumesh_dtu_scan63.yaml, and the code takes a mean of squared differences over all components trainer.py.
• Mask loss. Eq. 5 has no mask term eq. 5. The code adds a binary cross-entropy term on object masks with weight 0.1 trainer.py neumesh_dtu_scan63.yaml. The supplement describes this term app. B.1.
• Sign weight ω^n. The paper fixes it at 0.1 §3.1. The code uses 0.1 as a constant by default neumesh.py. A learnable weight exists but is off by default [code: models/frameworks/neumesh/init.py#L47-L49] neumesh.py. This matches the paper's stated setting.
• Sharpness s. The paper uses Φ_s but gives no value or training rule §3.1. The code sets s = exp(ln_s × speed_factor) neumesh.py. It copies ln_s and the speed factor from the NeuS teacher [code: models/frameworks/neumesh/init.py#L84-L87]. The DTU config sets the speed factor to 10 neumesh_dtu_scan63.yaml. The code freezes ln_s during training train.py neumesh.py.
• Painting iterations. The supplement gives 8000 app. B.3. The example painting config uses 10000 dtuscan105_whiteglass.json. The learning rate of 0.01 matches paint.py.
• Learning-rate floor. The paper gives no minimum learning rate. The code decays to 10% of the peak base.py.
• Training length. The paper gives no iteration count. The DTU config uses 200000 neumesh_dtu_scan63.yaml.
• Unused loss class. DensityLoss is defined and used nowhere in the training path trainer.py trainer.py.
• Sign vectors learned. The paper says the sign vectors are learned §3.1. The code makes them trainable parameters neumesh.py, optimised with the rest of the model base.py.
• Mesh extraction. The paper's 256 voxels over [-1, 1] app. A match the defaults extract_mesh.py.
• Nearest-neighbour count and decoder sizes. K = 8 app. A matches mesh_grid.py. Decoder depths match the paper's 3 and 4 hidden layers neumesh.py.
• Train/test split. The paper holds out 10% of DTU images §4.1. The DTU config uses split: train with a downscaled validation set neumesh_dtu_scan63.yaml. The 10% ratio is not verified in the code.

Pitfalls

• Normal orientation. The sign vectors start at the normals computed by Open3D mesh_grid.py. The paper does not say how those normals are oriented. A flipped normal changes the inside/outside meaning of h, and the regulariser pulls the sign vectors back towards that same initial normal.
• Detached neighbour selection. The neighbour distances are detached, so the neighbour indices and inverse-distance weights get no gradient mesh_grid.py. Gradients flow through the codes, the sign vectors and the offset p_k.
• Epsilon in the weights. Inverse distances use a 1e-7 offset mesh_grid.py. A query point that lands exactly on a vertex gets a finite but very large weight, and the paper does not discuss this.
• Hardware and install. The code needs the FRNN CUDA library for nearest-neighbour search and an Open3D development build README.md. The mesh grid is built with K = 32 and a radius of 100 mesh_grid.py.
• Teacher dependency. A NeuS teacher must exist first, and the mesh is taken from its SDF. Training from scratch was reported to give artifacts §3.2.
• Eikonal cost. The Eikonal term needs the SDF gradient at every sample, which means double backpropagation neumesh.py. The paper gives no cost figure for this.
• Speed. Rendering takes about 30 s per view, from many network queries and neighbour searches app. C. The authors name coefficient caching, hash encodings and MVS priors as possible speed-ups app. C.
• Painting restrictions. Painting training uses only rays through a slightly dilated paint mask and random view directions §3.4. The dilation size is not given. The code draws each random direction from a uniform vector in the unit cube and normalises it, so the directions are not uniform on the sphere renderer.py.
• NeRF-synthetic setup. The README says to set intrinsic_from_cammat: True for hotdog, chair and mic README.md. The paper does not mention this.
• Mesh-level limits. A mesh that fails to reconstruct (smoke, liquid) cannot be represented §5. Mesh quality depends on the NeuS teacher app. C.
• Texture filling and decoders. A filled region keeps the template model's radiance decoder app. B.3. Mixing two decoders at boundaries needs a blend that the paper does not fully specify.
• Unstated averaging. tab. 1 does not say which NeRF-synthetic scenes are averaged, so cite the table values as given rather than per-scene numbers.

Lineage

• Builds on: NeuS as the teacher model and the source of the SDF-to-opacity rendering §3.1 §3.2. The paper also draws on NeRF for volume rendering and positional encoding §3.1. The distillation scheme follows KiloNeRF §3.2. The Eikonal term follows IGR §3.2.
• Compared against: NeuS and NeuTex §4.2.
• Superseded by: not stated in the content of this card.