CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation

In plain words

Describing an object in words is easy. Building a model that turns those words into a 3D shape is hard, because almost no data pairs sentences with 3D shapes. CLIP-Forge avoids needing those pairs. It turns each 3D shape into 2D pictures, which is easy for any shape, and it uses CLIP, a model that already places pictures and sentences in one shared space where matching ones sit close together. So the model can learn from pictures alone.

Training has two steps. First, a small autoencoder learns to squeeze each shape into a short list of numbers, its embedding, and to rebuild the shape from that list. Second, a model learns which embedding goes with which picture's CLIP features. To generate, you type a sentence. CLIP turns it into features in the same shared space as the pictures, the second model turns those features into a shape embedding, and the decoder from the first step turns that embedding into a shape. Because the second model is probabilistic, one sentence can give several different shapes, and nothing is optimised again for each request.

For an engineer the main point is that the model never sees a sentence paired with a shape during training. It is only as good as the match between picture and sentence features in CLIP, and the pitfalls below cover where that match breaks down.

TL;DR

CLIP-Forge generates 3D shapes from text without any text-shape pairs. A voxel autoencoder learns a 128-d embedding per shape, and a conditional RealNVP flow learns to produce those embeddings from frozen CLIP features of rendered views; at test time the CLIP text features take the place of the image features. Generation is feed-forward, with several shapes per prompt possible. The author-reported headline is FID 2425.25 and 83.33% category accuracy on 224 prompts, ahead of two supervised text2shape baselines. The released code has no licence, so this card uses only pseudocode from it, and several training details differ from the paper.

Problem

Text-to-image models learn from large paired datasets, but text-to-shape data are scarce. The only paired set the authors found, text2shape, has 56,399 descriptions and covers chairs and tables only §4. CLIP-based methods for 2D images work by iteratively optimising an image for each prompt, and the authors note that this does not transfer clearly to 3D, where the complexity is much higher §2. The goals are zero-shot generation over ShapeNet categories, no optimisation at inference, and several shapes for one prompt §1.

Method

The training data are shapes with no text labels. Each training item holds rendered images, a 32^3 voxel grid, query points and their occupancy values §3. Training has two stages, and inference runs the trained parts in sequence §3.3. The overview diagram is fig. 3.

Stage 1: shape autoencoder §3.1

The encoder reads the voxel grid and outputs a 128-d shape embedding §4 app. A. During training, Gaussian noise is added to that embedding eq. 1. The authors report that the noise improves generation and diversity in their ablation tab. 3. The decoder is an implicit occupancy model inspired by Occupancy Networks: it takes the embedding together with a query point and predicts the occupancy at that point §3.1. The loss is the mean squared error between predicted and true occupancy §3.1.

A point-cloud variant uses a PointNet encoder, a FoldingNet decoder and a Chamfer loss §3.1 app. A. The authors report only qualitative results for it §4.5.

# Stage 1 training step: voxel encoder and implicit decoder  [§3.1] [eq. 1]
# Input:  voxels (B, 32, 32, 32); query_points (B, T, 3); occupancy (B, T)
# Output: scalar loss

def autoencoder_step(voxels, query_points, occupancy):
    # Encode each voxel grid into a 128-d shape embedding  [§3.1] [app. A]
    shape_embs = voxel_encoder(voxels)                  # (B, 128)
    # Add Gaussian noise to the embedding, during training only  [eq. 1]
    shape_embs = shape_embs + 0.1 * random_normal(shape_embs)
    # Predict an occupancy value for each query point  [§3.1]
    predicted = implicit_decoder(query_points, shape_embs)  # (B, T), unbounded
    # Mean squared error against the ground-truth occupancy  [§3.1]
    loss = mean((predicted - occupancy) ** 2)
    return loss

Stage 2: conditional normalising flow §3.2

After stage 1, every training shape has an embedding e_n. For each rendered image I_n, a frozen ViT-based CLIP image encoder produces a condition vector c_n eq. 2 §3.2. The prior is a RealNVP flow with five coupling blocks §3.2 app. A. In every coupling layer, the condition is concatenated to the input of the scale and translation networks eq. 2 eq. 3. The authors compare two masks: checkerboard, which the paper defines by spatial coordinates, and dimension-wise, which keeps the first half of the vector §3.2. The loss is the exact log-likelihood under a unit Gaussian base distribution, including the log-determinant of the Jacobian §3.2. This density equation is unnumbered in the paper.

# Stage 2 training step: conditional RealNVP prior  [§3.2] [eq. 2] [eq. 3]
# Input:  shape_embs (B, 128); image_feats (B, 512), L2-normalised  [§3.2]
# Output: scalar negative log-likelihood

def prior_step(shape_embs, image_feats):
    # Map each embedding to a latent, conditioned in every layer  [§3.2]
    latents, log_jacobian = flow_forward(shape_embs, image_feats)
    # Log-density under a unit Gaussian, summed over dimensions  [§3.2]
    base_log_prob = sum(normal_log_pdf(latents), axis=-1)  # (B,)
    log_prob = base_log_prob + log_jacobian                 # (B,)
    return -mean(log_prob)

# One affine coupling layer  [eq. 2] [eq. 3]
# mask: 1 where values pass through unchanged, 0 where they are transformed

def coupling_forward(values, cond, mask):
    # Only the unchanged part and the condition feed the two networks  [eq. 2]
    net_input = concat(values * mask, cond)             # (B, D + 512)
    # Log-scale and shift, zeroed where the mask is 1  [eq. 3]
    log_scale = scale_net(net_input) * (1 - mask)
    shift = translate_net(net_input) * (1 - mask)
    # Affine map applied to the transformed part  [eq. 3]
    out = values * exp(log_scale) + shift
    # The Jacobian of this layer is triangular, so its log-determinant is
    # the sum of the log-scales  [§3.2]
    log_det = sum(log_scale, axis=-1)
    return out, log_det

Inference: text to shape §3.3

The text is encoded by the CLIP text encoder, and its feature vector is used as the condition §3.3. The flow is run in reverse on latent noise drawn from a unit Gaussian, which gives shape embeddings. The implicit decoder then gives an occupancy value at every query point, and a threshold turns those values into a shape §3.3 app. E. The paper uses the mean of the Gaussian, which is zero noise, as a prototype shape for each prompt §3.3.

# Inference: text to shapes  [§3.3]
# Input:  text; decoder; flow; num_samples
# Output: boolean occupancy grids (num_samples, 32, 32, 32)

def text_to_shapes(text, decoder, flow, num_samples):
    # Encode the prompt and normalise it, as with the image features  [§3.3]
    text_feat = normalise(clip_text_encoder(text))      # (512,)
    # Latent noise from a unit Gaussian; one sample may be zero, which
    # gives the mean embedding  [§3.3]
    noise = random_normal(num_samples, 128)             # (num_samples, 128)
    # Invert the flow under the text condition to get shape embeddings  [§3.3]
    shape_embs = flow_inverse(noise, text_feat)         # (num_samples, 128)
    # Evaluate the decoder on a 32^3 grid of query points  [§3.3] [app. A]
    grid = make_grid(32)                                # (32768, 3)
    occupancy = decoder(grid, shape_embs)               # (num_samples, 32768)
    # Keep points above the threshold; 0.05 for metrics  [app. E]
    return reshape(occupancy > 0.05, (num_samples, 32, 32, 32))

Hyperparameters

Values marked "code" come only from the released code. The paper states nothing about them.

Results

Author-reported. The table compares CLIP-Forge with two text2shape baselines, both trained on text2shape chairs and tables with the same CLIP text encoder and occupancy decoder §4. The evaluation uses 224 prompts, 32^3 voxels, FID against ShapeNet test shapes with a 13-class voxel classifier, MMD by IoU matching, and Acc. as the classifier's agreement with the prompt's label §4.

Human study: nine crowd workers per pair chose the shape generated from the detailed prompt in 70.83% of pairs, where the alternative was the generic category prompt §4.3 app. J. Attribute prompts were identified more easily than sub-category prompts §4.3.

Appendix comparison with supervised models app. B:

Category-wise accuracy of the main model app. C:

Prompt prefix tab. 2, with the main setting in the first row:

Stage 1 design choices tab. 3; the final configuration is the row with noise, latent 128, VoxEnc and RN-OccNet:

Stage 2 prior design tab. 4:

Number of renderings and CLIP backbone tab. 5:

The paper attributes the gap between ablation and main numbers to seed selection: the ablations report means over three seeds, and the main section reports the best seed app. A. For example, the RealNVP-D row in tab. 4 gives FID 2591.87, while the main setting in tab. 1 gives 2425.25.

Qualitative results: several shapes for one prompt fig. 4; category, sub-category, common-name and attribute prompts fig. 5; interpolation between two prompts, which the authors read as a smooth conditioning space fig. 7; point clouds fig. 8. The authors also show that a text2img-then-img2shape pipeline gives poorer shapes app. D. Out-of-distribution prompts such as "a shoe" or "a human" give shapes from the training categories app. F.

Paper vs code

The paper's title block links the repository. It sits under the AutodeskAILab GitHub organisation, and the authors are at Autodesk (title block). Pinned commit ec2eb15004c544bc4e3f8e5aee14f2f9eef477ae. The repository has no LICENSE file at the top level; the only license file is under utils/libmcubes, which is vendored third-party code. The GitHub licence API returned 404. No verbatim code is quoted in this card.

• Stage 2 learning rate. The paper gives 1e-4 for both stages §4. The code uses Adam at 3e-5 for the flow train_post_clip.py.
• Stage 2 epochs. The paper gives 100 §4. The code's default is 300 and the README does not override it train_autoencoder.py [README.md#L49-L60].
• Learning-rate schedules. Both training scripts build a cosine schedule [code: train_autoencoder.py#L396] train_post_clip.py, but neither calls scheduler.step(), so the rate stays constant. The paper says nothing about a schedule.
• Weight decay. Stage 1 uses Adam with weight decay 1e-5 [code: utils/helper.py#L72]. The paper does not mention it.
• Checkerboard mask. The paper defines it by spatial coordinates §3.2. The code applies it by the parity of the vector index, since the embedding is a plain vector latent_flows.py.
• Ablation masks. The stage 1 ablation text says checkerboard masking was used for most experiments §5, but the main section and the code use dimension-wise masking §4 train_post_clip.py.
• Decoder input. The paper says the embedding and the query point are concatenated §3.1. The code adds two linear projections of them [code: networks/autoencoder.py#L195-L198]. For a plain linear layer these are equivalent; this equivalence is my reading, not stated in the paper.
• Decoder variants. The ablation includes a CBN-OccNet decoder and a residual voxel encoder tab. 3. The released get_model always builds the plain voxel encoder and the residual occupancy decoder, and ignores decoder_type [code: networks/autoencoder.py#L229-L242].
• Stage 2 noise. The paper adds noise only in stage 1 eq. 1. The code also adds 0.1 times a standard normal to the stage 2 embeddings during training [code: train_post_clip.py#L209-L210].
• Noise scale. The paper writes N(0, 0.1) eq. 1. The code uses a standard deviation of 0.1 autoencoder.py; if the paper's second argument is a variance, the scales differ by about 3.2 times.
• Output range. The decoder has no output sigmoid by default [code: networks/autoencoder.py#L166] autoencoder.py. The threshold is therefore applied to an unbounded value.
• Test-time noise clipping. The test script clips the sampling noise to [-1, 1] test_post_clip.py. The paper does not mention clipping, and the training-time visualisation does not clip train_post_clip.py.
• Mean embedding. The code uses zero latent noise test_post_clip.py, which matches the paper's mean of the Gaussian §3.3. It is the flow's image of zero noise, not the average of training embeddings.
• Views. The paper's best setting is 20 renderings tab. 5. The code default is 5 train_post_clip.py, and the README example uses 1 [README.md#L50].
• FID features. The paper says the feature vector comes from the "fourth layer" of the classifier §4. The code uses the 512-d output of the voxel encoder [code: classifier.py#L43] classifier.py.
• MMD. The paper defines MMD on IoU matching §4. No MMD code appears in the repository.
• Evaluation grid. The stage 1 validation IoU samples voxel centres [code: train_autoencoder.py#L170-L173]. The inference grid is a linear spacing from -0.5 to 0.5 inclusive test_post_clip.py, so its points are not the voxel centres. The paper does not state the grid.
• Image preprocessing. Not stated in the paper. The code resizes to 224 with bicubic interpolation, centre-crops and applies CLIP normalisation [code: dataset/shapenet_dataset.py#L172-L177].
• Point cloud. The README says the point-cloud code is "semi done" and has not been tested, and no point-cloud pretrained models are released [README.md#L88-L90]. The paper shows only qualitative point-cloud results §4.5.
• Pretrained weights. Checkpoints, the classifier and the data come from an external archive [README.md#L36-L39]. Its licence is not stated, and this card does not rely on it.

Pitfalls

• Batch-norm running statistics. The flow's batch-normalisation layer has momentum 0 by default latent_flows.py. Its running statistics therefore equal the most recent training batch, and inference uses them [code: networks/latent_flows.py#L42-L48] latent_flows.py. The paper mentions batch normalisation in the flow but does not describe these statistics app. A. Any change, such as a true running average, departs from the released code and should be reported as such.
• Threshold sensitivity. Metrics use 0.05, but the authors chose a per-category threshold by grid search for the figures app. E, and the README advises tuning it [README.md#L84]. The appendix shows that the threshold visibly changes the shapes app. E.
• Unbounded decoder output. The output is not squashed to [0, 1] [code: networks/autoencoder.py#L166], so the threshold of 0.05 is tied to this model and training run.
• Modality gap. The flow is trained only on image features, and text features replace them at inference §3.3. The paper does not measure how well the two match. It attributes the weaker categories to data imbalance, not to the text-image match app. C. The prompt prefix also moves FID between 2400 and 3142 tab. 2.
• Sampling noise range. The test code clips the sampling noise to [-1, 1] [code: test_post_clip.py#L202], which the paper does not mention (see Paper vs code). Sampling without the clip gives different outputs from the released test script.
• Cost of inference. The paper stresses that no optimisation is needed §1. Each sample still needs the flow inversion and 32768 decoder evaluations per shape, one per grid point test_post_clip.py test_post_clip.py.
• Epoch-trigger quirk. The visualisation condition (epoch + 1) % 5 == True is true only when the remainder is 1, because True equals 1 in Python [code: train_post_clip.py#L355-L357] train_autoencoder.py. It affects only when figures are written, not training.
• Checkpoint selection. Stage 1 keeps the checkpoint with the best validation IoU at threshold 0.05 train_autoencoder.py [code: train_autoencoder.py#L187]. Stage 2 keeps the best validation loss [code: train_post_clip.py#L366-L370].
• Views and precomputation. Stage 2 precomputes CLIP features for num_views random renderings per shape train_post_clip.py. Validation uses view 0 shapenet_dataset.py. Training memory and time grow with the number of views.
• Scope. The model is trained only on the 13 ShapeNet categories, and it generalises poorly to other categories app. F §6. It produces geometry only, so colour words in long prompts are ignored app. H §6.
• Prompt counts. The paper says 234 prompts are defined and 224 are used for FID and MMD §4. The reason for the difference is not given.
• Generic shapes. Some prompts give shapes that the crowd could not tell from the category prompt app. J, for example "a laboratory bench" §4.3.

Lineage

• Builds on: CLIP, for the shared text-image embedding (https://arxiv.org/abs/2103.00020) §3.2; RealNVP, for the coupling flow (https://arxiv.org/abs/1605.08803) §3.2; Occupancy Networks, for the implicit decoder §3.1; PointNet and FoldingNet, for the point-cloud variant §3.1.
• Compared against: text2shape, a supervised text-to-shape model trained on paired chairs and tables tab. 1 §4.
• Superseded by: not stated in the sources.