Zero-Shot Text-Guided Object Generation with Dream Fields

In plain words

Dream Fields makes a 3D object from a sentence. It starts from a randomly initialised 3D cloud of density and colour, and repeatedly "photographs" it from random angles all the way around. A model that was trained to judge how well a picture matches a caption looks at each photo, and the cloud is nudged so that the photos match the caption better. Nobody had to show the system any 3D models, which is the usual bottleneck.

Left alone, this process makes messy shapes: thin floating haze, and a volume that fills the whole camera view instead of a single object. The authors fix this with a few general rules. They reward the cloud for being mostly see-through, put random backgrounds behind it during training so it cannot hide in the background, keep it inside a fixed box, and use a slightly different neural network design.

The result is a single object per caption, viewable from many angles. It is an optimisation that takes time per object, not a fast generator.

TL;DR

• Text-to-3D without 3D training data: a NeRF is optimised so that renders from random azimuths match a caption under a frozen CLIP image-text model §4.2.
• Three priors make it work: a transmittance (sparsity) loss, random background augmentation and scene bounds, plus a residual MLP with Swish and softplus density §4.5 §4.6 §4.7.
• Author-reported CLIP R-Precision reaches 68.3 with 20K iterations at 252 px renders, against 31.4 for a simplified NeRF baseline tab. 1.
• Each object takes about 1 h 12 min for 10K iterations on 8 TPU cores, at an author-estimated $3-4 per object app. C.

Problem

Detailed 3D models are expensive to author by hand §1. Learned 3D generators trained on 3D datasets cover only a few categories such as ShapeNet, because captioned 3D data is scarce §1 §2. Text-to-image models have no 3D output §2. The paper asks whether a text-image model can drive 3D generation for open-set objects without 3D or multi-view data §1.

Naively optimising a NeRF with only a CLIP loss from a few viewpoints gives severe artifacts: floating semi-transparent density, near-field occlusions, and a volume that fills the view §4.3 fig. 3.

Method

Overview. A Dream Field is a NeRF trained with a CLIP-based caption loss instead of photos §4 fig. 1. Each iteration samples a camera pose, renders an image, and minimises the caption loss with respect to the MLP weights §4.2.

Object representation §4.1. An MLP with parameters θ maps a 3D point x to a density σ_θ(x) and a colour c_θ(x). The field does not depend on viewing direction, which the authors found did not help.

Rendering uses the volume-rendering quadrature. Transmittance is the probability that light is not absorbed between the near bound and a point §3.1 eq. 1 eq. 2. The discrete form uses N segments per ray, jittered, with N = 192 during optimisation §4.1 eq. 3 eq. 4.

Input encoding §3.1. The position is encoded by mip-NeRF's integrated positional encoding, combined with random Fourier features whose frequencies are sampled as ω = 2^u · d with u ∼ U[0, L] and d ∼ U(S²), using L = 8 eq. 5 eq. 6 app. C.

Caption objective §4.2. A frozen CLIP image encoder g and text encoder h map renders and captions into a shared, normalised space. The loss is the negative similarity, averaged over sampled poses:

\mathcal{L}_\mathrm{CLIP}(\theta, \mathbf p, y) = -\,\mathbf g(\mathbf I(\theta, \mathbf p))^\top \mathbf h(y) eq. 7

The default image-text model is CLIP ViT-B/16 tab. 1 caption §5.2. The paper also uses a LiT ViT-B/32 model trained on a larger web set §3.2 §5.2.

Camera sampling §4.4. Azimuth is sampled uniformly over 360°, so each iteration sees a different side. The elevation is fixed at 30° above the equator, the radius at 4, and the focal length is 1.2× the base value to enlarge the object by 20%. The authors report that narrow azimuth intervals produce flat, billboard-like geometry §4.4 fig. 8.

Background augmentation §4.5 fig. 4. The render is composited over a random background, chosen from uniform noise, a random-colour checkerboard, or a random smooth Fourier texture. Each background is blurred with a random standard deviation. Training renders are then randomly cropped and resized before CLIP. At test time the background is white fig. 4.

Transmittance regulariser §4.5. Let T be the per-ray transmittance and τ a target. The regulariser rewards the mean transmittance for rising up to τ:

\mathcal{L}_\mathrm{Tr} = -\min\left(\tau, \operatorname{mean}\left(T(\theta, \mathbf p)\right)\right) eq. 8

\mathcal{L}_\mathrm{total} = \mathcal{L}_\mathrm{CLIP} + \lambda\, \mathcal{L}_\mathrm{Tr} eq. 9

τ = 88% in the experiments, annealed in from 40% over the first 500 iterations §4.5. The weight λ is not stated in the paper. The authors also compare a multiplicative gated form and a Neural Volumes beta prior in the ablation §5.4 tab. 2 eq. 10 eq. 11.

Scene bounds and origin tracking §4.6 app. C. Density outside a cube of side 2 is masked. The scene origin is tracked as an exponential moving average (decay 0.999) of the density centre of mass, and rays are shifted accordingly, so the object need not sit at the centre.

Architecture §4.7 app. C. Instead of the NeRF MLP, the authors use a residual MLP with a residual connection around every two dense layers, LayerNorm at the start of each residual block, a bottleneck feature dimension, Swish activations, and a softplus density. It has 280K parameters against 494K for NeRF §4.7.

Optimisation app. C. Adam with ε = 1e-5. The learning rate warms up exponentially from 1e-5 to 1e-4 over 1500 iterations and is then held constant.

# Dream Fields optimisation loop, reconstructed from [§4.2] to [§4.7] and
# [app. C]. The paper has no numbered algorithm.
# Input:  caption y; frozen image encoder g and text encoder h
# Output: MLP parameters theta
def train_dream_field(caption, iters):
    # Embed the caption once with the frozen text encoder [§4.2]
    text_embedding = normalize(h(caption))
    # Initialise the MLP with Flax defaults [app. C]
    theta = init_mlp()
    for step in range(iters):
        # Sample a camera: azimuth uniform in 360 deg, elevation 30 deg,
        # radius 4, focal length 1.2x the base value [§4.4] [app. C]
        pose = sample_pose()
        # Render 168 px with 192 jittered samples per ray [§4.1] [app. C]
        rgb, acc = render(theta, pose, num_samples=192)
        # Composite over a random background so density cannot hide
        # behind it [§4.5] [fig. 4]
        background = random_background()
        image = rgb * acc + background * (1 - acc)
        # Crop to 154 px and resize to CLIP's 224 px input [app. C] [§5.4]
        view = resize(random_crop(image, 154), 224)
        # CLIP similarity loss on normalised embeddings [eq. 7]
        loss_clip = -dot(normalize(g(view)), text_embedding)
        # Transmittance regulariser: push mean transmittance up to tau
        # [eq. 8]
        loss_trans = -min(tau(step), mean(1 - acc))
        # Combined objective; lambda is not stated in the paper [eq. 9]
        loss = loss_clip + lam * loss_trans
        # Adam update with the learning-rate schedule [app. C]
        theta = adam_update(theta, grad(loss, theta), lr(step))
    return theta

def tau(step):
    # Target transmittance: 40% rising to 88% over 500 iterations
    # [§4.5]. The interpolation form is not stated in the paper.
    return anneal(step, 500, start=0.40, end=0.88)

def lr(step):
    # Exponential warm-up from 1e-5 to 1e-4 over 1500 iterations,
    # then held constant [app. C]
    if step < 1500:
        return exp_interpolate(1e-5, 1e-4, step / 1500)
    return 1e-4

Positional-encoding and render settings for the quantitative runs are in the hyperparameter table. The figures named above show the effect of each component fig. 5 fig. 7 fig. 9.

Hyperparameters

Results

Author-reported. Setup: object-centric COCO captions, 153 test captions, 2 seeds per caption, R-Precision computed on renders from a held-out pose with CLIP ViT-B/32 and LiT B/32 §5.1 §5.2 tab. 1. Baseline: a simplified NeRF with the same training recipe.

The sparsity ablation compares no regulariser, σ perturbation (47.71), a beta prior (50.33), a gated loss (34.64) and the clipped additive loss (62.09) tab. 2. The model ablation scores each optimised model under three retrieval models tab. 3. Italic cells use the same model for optimisation and retrieval, and the paper calls them overfitting.

Paper vs code

Code: official repository google-research/google-research, subdirectory dreamfields/, at commit 3ef52111c1fabc301880818f5a547570e926a76b, Apache-2.0. The paper's project page (ajayj.com/dreamfields) links this code. Paths below are relative to the repository root. The repository was cloned to raw/code/2112.01455.

• Target transmittance. The paper uses τ = 88% annealed from 40% over 500 iterations §4.5. The code anneals the opacity target from 0.5 to 0.1, i.e. transmittance from 50% to 90%, exponentially in log space config_base.py schedule.py. The transmittance penalty uses min(1 - acct, 1 - acc_mean) lib.py.
• λ. The paper does not state λ. The code sets acc_lam = 0.5 config_base.py.
• Fourier frequencies. The paper gives ω = 2^u · d eq. 6. The code draws 128 random directions and radii, then casts the frequency vectors to int32, which truncates them to integers mipnerf.py. It also concatenates the three axis-aligned identity features mipnerf.py. The basis is fixed by seed 124124 and does not change over training mipnerf.py. The paper does not mention any of this.
• Integrated positional encoding annealing. The paper does not mention it. The code scales the IPE attenuation by max(1, 2^(1 - step/1000)), so it is coarser for the first 1000 iterations lib.py. The config comment says the annealing is disabled by decay_start = 1 config_base.py. The comment and the code do not agree; the code is what runs.
• Scene bounds. mr0 = mr1 = 1, so the bound ramp is inert and the cube is fixed config_base.py.
• Colour output. The code applies a sigmoid to the colour outputs mipnerf.py. The paper does not state the colour activation.
• Background blur. The paper gives no range for the blur σ. The code draws it from U[0, 10] with a 15-pixel kernel config_base.py augment.py.
• Augmented views per step. The code uses n_local_aug = 8 random views per device per step config_base.py. The paper does not state a view count.
• Test samples. The paper uses 512 samples at test app. C. The default test config uses 256 and the HQ config 512 config_base.py. I did not trace which one the paper's R-Precision evaluation uses.
• Learning rate. The paper's exponential warm-up matches schedule.py. After warm-up the code decays linearly to lr2; the default lr2 = 1e-4 makes this constant schedule.py config_base.py.
• Matches. Near/far 4 ∓ √3, focal multiplier 1.2, azimuth 0-360°, radius 4, 192 samples, CLIP ViT-B/16 default, EMA origin decay 0.999, Adam ε = 1e-5, and the beta prior disabled (beta_after = 2 · iters) all agree with the paper config_base.py config_base.py config_base.py config_base.py lib.py. The CLIP embeddings are normalised as the paper's notes suggest helpers.py.
• Elevation sign. The config uses phi = -30, and a code comment reads the negative as 30° elevation scene.py. This matches the paper.
• Not traced. The voxel and 2D baselines of app. D, the LiT model wiring, and the scaling-law claim in §4.5 (1 − τ ∝ f²/d²) are not checked in the code. The experiments/ directory is not reviewed.

Pitfalls

• Schedule mismatch. Paper: τ 40% → 88%. Code: transmittance 50% → 90%. An implementer following the text gets a different early-training regime. Interpolation is exponential in the code and unstated in the paper.
• λ is unstated. Results depend on the transmittance weight. The code uses 0.5. The paper says the gated variant is "sensitive to hyperparameters" §5.4.
• Duplicate values in the tables. In tab. 1, the rows "+ random crops" and "+ higher freq. Fourier features" both have LiT B/32 R-Precision 10.4575 ± 1.7522. In tab. 2, the clipped gated and clipped additive rows are identical (62.0915 ± 3.9352). These may be copy errors, so check them before citing a single cell.
• Non-monotonic table. In tab. 1, "+ track origin" (59.80) is below "+ scene bounds" (65.36) on CLIP B/32. The cosine-similarity column is marked NaN with a TODO. The text does not discuss the drop.
• Italic cells. The text says "CLIP B/32 performs the best" §5.4. The supporting cells in tab. 3 are italic same-model scores, which the paper itself labels as overfitting. The non-italic comparison does not show that ranking.
• Retrieval model shift. Optimisation uses one CLIP model and evaluation uses another. Scores therefore measure cross-model agreement as well as caption fit. The paper states this as a limitation for evaluation §5.1.
• Overfitting. Long runs keep lowering the training loss, but validation R-Precision falls after 5-10K iterations app. F fig. 13. Quantitative results stop at 10K; the 20K row is a separate setting.
• Fixed render resolution. Training renders are 168 px, cropped to 154 px, then upsampled to 224 px. The paper suggests the low render resolution may be why a larger CLIP patch size (B/32) suffices §5.4. High-resolution figures use 252 px.
• Camera-dependent artifacts. Narrow azimuth ranges give billboard-like geometry and artifacts from other angles §4.4 fig. 8. Using the same prompt from every pose can repeat patterns on several sides of the object §6.
• Cost. 10K iterations take about 1 h 12 min on 8 preemptible TPU cores app. C. The paper states that MLP inference and backpropagation through rendering dominate, not CLIP app. C. Memory use is not stated.
• Hyperparameter tuning. Values were tuned by hand on 74 development captions, mostly 20 of them app. C, and then shared across all scenes. Results on other prompt styles are not reported.
• Scene realism. Some geometric details are unrealistic: stalks attached to the shell, a blurry vase, and floating text from typographic attacks §5.3 app. F. The paper says the image-text model inherits harmful biases §6.
• Unstated details. Rays per step, batch size, the number of random Fourier features (128 in code), and background probabilities are not stated in the paper. Dropout is 0 by default in the code helpers.py.
• Licensing. The code is Apache-2.0 and may be cited in short snippets. No snippet is quoted here.

Lineage

• Builds on: NeRF (Mildenhall et al.) for the volumetric representation §3.1; mip-NeRF's integrated positional encoding §3.1; random Fourier features §3.1; CLIP, arXiv:2103.00020, for the caption score §3.2; DeepDream-style image optimisation §2; the differentiable parameterisation of Mordvintsev et al. for the gated loss §4.5.
• Compared against: DietNeRF, a semantic-consistency regulariser that needs real photos §4.2; CLIP-Forge, Text2Shape, and ClipMatrix, which need 3D or shape-class data §2; a voxel grid baseline app. D.
• Superseded by: none in this wiki yet.