MaPa: Text-driven Photorealistic Material Painting for 3D Shapes

In plain words

Game engines and film renderers do not show a mesh with a flat picture wrapped around it. They show it with a material: how shiny it is, how rough, what colour, what small scratches or patterns. Most text-to-3D tools produce only a colour texture, and that texture is hard to change later.

MaPa asks for a material for each part of a 3D shape, given a text prompt such as "a dark brown leather sofa". It first cuts the shape into parts and renders one view of it, where each part is a labelled region. A 2D image model then paints that view, so each part gets its own look. Parts that look alike are grouped. For each group, the method picks a starting "material recipe" from a small library and adjusts its settings until a rendering of it matches the painted image.

The output is a procedural material graph, a recipe of image operations with settings. Because the recipe is kept rather than a fixed picture, a designer can open it in modelling software and change it. The authors also show a text-driven way to add patterns to a finished material.

TL;DR

MaPa generates materials, not plain textures, for a mesh from a text prompt. A segment-conditioned ControlNet renders one view, GPT-4V labels each segment's material class, and each group's procedural material graph is retrieved and then fitted by differentiable rendering. The graphs stay editable. The author-reported FID and KID beat three texture baselines on chairs and ABO objects, and the user study favours MaPa on quality and prompt fidelity.

Problem

Texture-only methods (TEXTure, Text2tex) output a colour map and do not model how the surface reacts to light, so the result often looks dirty or inconsistent when lit §1, §4.2. Fantasia3D models material with a per-point network, which the authors call unstable and hard to edit §1. Training a model that maps text and meshes directly to material graphs would need a large set of paired meshes, material graphs and text, which does not exist §1. MaPa uses a pre-trained 2D diffusion model as the bridge instead §1, §3.

Method

Overview: the input mesh is split into segments, one view is rendered as a segment mask, a diffusion model paints that view, the painted parts are grouped, and each group gets a fitted material graph. Parts still without a material are filled in by repeating the process on neighbouring views §3, §3.4. Editing is a separate, optional step §3.5.

Step 1: segment-controlled image generation §3.1.

• The mesh is oversegmented. The default segments are its connected components; watertight meshes use a graph-cut decomposition §3.1, §4.4.
• Candidate views sample 360° of azimuth at a fixed 25° elevation and radius 2 units §3.1. The number of candidate views is not stated.
• The start view is the one with the most projected 2D segments. Ties go to the view with the largest projected mesh area §3.1.
• The 2D segments are sorted by area, numbered from large to small, and form the segmentation mask; background is 0 [§3.1, supp.: segment-conditioned ControlNet].
• A SAM-conditioned ControlNet, fine-tuned by the authors, renders the RGB image from this mask eq. 1 §3.1. The authors use ControlNet with SAM masks instead of depth-to-image diffusion, which they say gives multi-colour segments §3.1.

\{\hat{s}_i\}_{i=1}^{M} = \mathcal{P}(\{s_i\}_{i=1}^{N})

z_{t-1} = \epsilon_\theta(z_t, t, \{\hat{s}_i\}_{i=1}^{M}) eq. 1

Step 2: material grouping §3.2.

• Segments are nodes. Two segments are connected when they have the same material class and similar colour. Each connected component is one group §3.2.
• Material class: GPT-4V, given the generated image and a fixed prompt [supp.: material classification]. Eight classes: wood, metal, plastic, leather, fabric, stone, ceramic, rubber §4.1.
• Colour: an albedo network removes shading from the generated image. It is a Stable Diffusion model conditioned on CLIP image embeddings, fine-tuned on ABO material data to map RGB to albedo §3.2, supp.: albedo estimation.
• Two segments are similar when the CIE distance between their median albedo colours is below λ = 2 §3.2.

Step 3: material graph selection and optimisation §3.3.

• A library of procedural material graphs is pre-rendered as thumbnails under eight environment maps §3.3.
• Selection: crop the group region, compare it by CLIP cosine similarity to the same-class graphs, and take the best match §3.3.
• Groups with an area below 1000 pixels get a homogeneous material instead of a graph §3.3.
• Optimisation: a differentiable renderer with DiffMat v2 converts the graph to texture maps (albedo, normal, roughness), samples them with UV maps from Blender, and renders with a microfacet BRDF §3.3 eq. 2.
• Lighting: a 2D spatially varying light field is initialised by InvRenderNet. A residual light, stored in a 2D hash grid with a shallow MLP and initialised to zero, is added and passed through a ReLU §3.3.
• Loss: an albedo term matches the rendered albedo to the predicted albedo, and a render term combines masked L1 with a VGG Gram-matrix style loss against the generated image. The total is the sum of the two [supp.: material graph optimization].
• Before optimising, the UV scale and rotation are chosen by searching scales 0.5 to 8.0 and orientations of −45°, 0°, 45° and 90° for the lowest Gram loss [supp.: material graph optimization].

A_{uv}, N_{uv}, R_{uv} = \mathrm{DiffMat}(G)

A, R = S(A_{uv}, R_{uv})

N = \mathrm{Rot}(S(N_{uv}))

I_{rend} = \mathcal{R}(A, N, R, L) eq. 2

\mathcal{L}_{albedo} = \|\mu(A \cdot M) - \mu(A^* \cdot M)\| [supp.: material graph optimization]

\mathcal{L}_{render} = \mathcal{L}_{L1} + \mathcal{L}_{style}, \quad \mathcal{L} = \mathcal{L}_{albedo} + \mathcal{L}_{render} [supp.: material graph optimization]

Step 4: iterative material recovery §3.4.

• Parts with no material yet are inpainted on the next view. The next view is the view adjacent to the start view §3.4.
• A segment needs a material when it has none, or when its projected area in the previous view is less than a quarter of its area in the current view §3.4.
• The previous and current views are rendered with the materials assigned so far, and their images are concatenated as input to a SAM-conditioned inpainting network. Only segments needing a material are inpainted §3.4.
• Step 3 runs again on the new groups. A segment that shares a group with an already-assigned segment copies that graph §3.4.
• The loop repeats until every region has a material §3.4.

# MaPa: material painting for one mesh  [§3]
# Input:  mesh, text prompt
# Output: one material graph per segment group

def paint_materials(mesh, prompt):
    # Split the mesh into segments; the default is connected components  [§3.1]
    segments = split_into_segments(mesh)
    # Start view: most projected segments, ties broken by projected area  [§3.1]
    view = pick_start_view(segments, candidate_views)
    # Segment mask for the start view, numbered by area, background 0  [§3.1]
    seg_mask = project_segments(segments, view)
    # Segment-conditioned ControlNet paints the view  [§3.1] [eq. 1]
    image = controlnet_generate(prompt, seg_mask)
    # Group segments by GPT-4V class and albedo colour (CIE, lambda = 2)  [§3.2]
    groups = group_segments(segments, image)
    for each group in groups:
        # Retrieve the best library graph by CLIP similarity  [§3.3]
        graph = retrieve_graph(library, crop(image, group))
        # Fit graph and residual light to the image  [§3.3] [eq. 2]
        graph = optimize_graph(graph, image, group)
    # Repeat on adjacent views until every segment has a material  [§3.4]
    while not all_segments_assigned(segments):
        view = next_adjacent_view(view, candidate_views)
        # Inpaint only the segments that still need a material  [§3.4]
        image = inpaint_unassigned(segments, view)
        groups = group_segments(segments, image)
        for each group in groups:
            graph = retrieve_graph(library, crop(image, group))
            graph = optimize_graph(graph, image, group)
    return graphs_per_group

Step 5: downstream editing §3.5.

• The graphs can be opened and changed by hand in Substance Designer §3.5.
• High-level edits are text-driven. GPT-4 receives the user's instruction, the material type and a set of local filter functions with examples, and writes a Python command. The user runs that command locally [§3.5, supp.: downstream editing]. The paper lists twelve such functions, for example add dust, add dirt, add scratches [supp.: downstream editing].

Hyperparameters

Results

Author-reported. Setup: test set of 300 chairs from TMT, plus 30 random ABO objects; 10 views each §4.2. Metrics are FID and KID (lower is better) against real images, and a user study on overall quality and fidelity to the prompt, on a 1–5 scale, with 40 raters (16 professional artists, 24 members of the public) §4.2. Baselines are TEXTure, Text2tex and Fantasia3D, all rendered with a diffuse-colour pass in Blender for the first two §4.2.

Other results are qualitative only:

• Watertight meshes are painted after graph-cut segmentation §4.4.
• Diversity: different seeds give different materials, and the renders match the generated images §4.3.
• Image-prompt appearance transfer uses the IP-Adapter image encoder §4.3.
• Editing a bag with text prompts §4.4.
• Timing: MaPa takes 7 minutes on average per shape. Without grouping it takes 33 minutes. The dataset behind the timing is not stated §4.4.
• Ablations ("w/o Albedo", "w/o Grouping", "w/o Opt", "w/o Res", and the depth-to-image variant in the supplement) are shown as figures without numbers §4.4 [supp.: segment-controlled image generation].

Paper vs code

No official code.

Searched: (1) the paper's links. The metadata abstract names zju3dv.github.io/MaPa; the source names zhanghe3z.github.io/MaPa, which returned 404 at check time. The zju3dv page links only a PDF release and a video. (2) The repository github.com/zhanghe3z/MaPa, owned by a user account that matches the project page. Its contents are index.html, toggle.js, assets and videos. It has no training code and no LICENSE. It is not treated as official code. (3) GitHub repository searches for MaPa material and PaintMat zhanghe3z found no official implementation.

The paper's method and settings are therefore checked against the paper only.

Pitfalls

• No code, so the hidden details are unresolved. Learning rates, the split of the 300 iterations across the three stages, the ControlNet and albedo training schedules, the hash-grid size and the CLIP model are not stated.
• λ and the colour space. λ = 2 is given without its colour formula. The paper cites CIEDE2000 but does not say the formula is CIEDE2000. Do not assume it.
• Median colour in the albedo image. It is not stated whether the median is taken over all pixels of a segment or only over masked pixels.
• GPT-4V dependency. Material classes come from a hosted model whose version is not stated. The paper says it works "empirically" and mentions replacing it later §5, supp.: material classification. Results will not be exactly reproducible.
• Retrieval-only library. The material graph library is not released. It holds 84 graphs in 8 classes (derived from the supplement table). The paper says most graphs lack nodes that could be optimised for spatial variation, so the complex looks in the bag example come from retrieved patterns §5, §4.4. Only the wood example shows that optimisation adds spatial variation §4.4.
• Loss weights and the style layers. The albedo, L1 and style terms are summed with weight 1, which is stated [supp.: material graph optimization]. The VGG layers used for the Gram loss are not stated.
• Iteration stop rule. The loop runs until every region is assigned §3.4. No maximum iteration count or fallback is stated for segments that never meet the area condition. The text does not say which earlier view counts as "previous" beyond the adjacent-view rule.
• Segments depend on the mesh. The method works on connected components. A mesh with unclear parts gets over-segmented, and grouping then depends on the ControlNet guidance, which may fail on complex objects §5.
• Domain gap. The paper says the albedo network was trained on synthetic images, while the diffusion output is natural-looking. It admits this can cause mismatches and shows a failure case, where unusual light in the diffusion output makes the albedo estimate wrong and the material dissimilar §5.
• Small evaluation. The ABO test uses 30 objects, so the ABO FID and KID rest on few objects. The chair set is 300 objects. The paper does not report variance, confidence intervals or the statistical test behind the user study.
• Real-image source for FID. The text says FID compares rendered images to "the real image" but does not name which real images serve as the reference.
• User study. Each rater gave "60 responses", but what was rated, and how the items were assigned, is not stated.
• Runtime comparison. The paper compares its 10-minute runtime with 15 minutes for Text2tex and 25 minutes for Fantasia3D. It does not say whether those were measured on the same machine.
• Commercial software. Editing needs Substance Designer. The paper treats the graphs as usable in existing commercial software, which is an external dependency.
• Section numbers. Equations and figures are cited by section, because the source numbering was not compiled to a PDF.
• Source note. The main file (main_paper_submit.tex) includes the experiments from 04_experiments_v2.tex. The supplement (06_supp.tex) is not included in that file, so its content comes from the separate supplement source. Whether the arXiv PDF includes it was not checked.

Lineage

• Builds on: procedural material graphs and DiffMat v2 (li2023end), the 2D spatially varying lighting model InvRenderNet (li2020inverse), material retrieval with CLIP as in PSDR-Room (yan2023psdr), and the SAM-conditioned ControlNet (gao2023editanything). The segment mask uses SAM (kirillov2023segment).
• Compared against: TEXTure (richardson2023texture), Text2tex (chen2023text2tex) and Fantasia3D (Chen_2023_ICCV).
• Superseded by: not stated.