Textual inversion

Overview

Textual inversion teaches a text-to-image diffusion model a new word for one concept. The only parameter trained is the embedding vector of a new token; the other embeddings and all model weights stay frozen. The learning signal is the diffusion denoising loss on the images that show the concept 2302.10663 §3.2 2302.10663 app. A.

The one compiled card covers a single use of the idea, in which one image replaces a set of photos. Other variants have no card yet, so this page cannot compare them.

Variants

Single-image textual inversion (RealFusion)

The input is one image of an object, with the background removed. The paper cannot get other views of the object, so it builds a small training set from random augmentations of that image. It then learns one token embedding with the diffusion loss, using the prompt "an image of a " [arxiv:2302.10663 §3.2]. The augmentations are a rotation of up to 10°, a random resized crop, colour jitter, occasional greyscale and blur, and a horizontal flip. The paper gives these as a listing in its appendix 2302.10663 app. A.

The paper's stated settings are 3000 steps, batch 16, learning rate 5e-4, weight decay 1e-2 with Adam, and 512 px images 2302.10663 app. A. The token can start from a random vector, from a word chosen by hand, or from the best-matching noun found with CLIP over WordNet. The paper used the manual start for its main figures and CLIP for its supplement [arxiv:2302.10663 app. A]. The learned token is then used inside the prompt of a score distillation loss on random views of a 3D model. That loss is the other half of the method 2302.10663 §3.2 and is covered on the Score distillation sampling page. The card for the full method is RealFusion: 360° Reconstruction of Any Object from a Single Image.

Trade-off: the paper says this step is its core contribution [arxiv:2302.10663 §3.2]. The card records no ablation that switches this step off, so its share of the reported results is not measured here.

Other variants

The original textual inversion paper, arXiv:2208.01618, has no card yet, so nothing from it is used on this page.

When to use which

The paper's reported comparison is for the whole RealFusion pipeline against Shelf-Supervised Mesh Prediction on 7 CO3D categories, not for this step on its own. On mean F-score the full method reports 9.58 against 8.24 [arxiv:2302.10663 tab. 1]. Treat that as the result of the pipeline.

Implementation sketch

Only the token embedding is trained. Everything else is frozen. The steps below follow the paper's appendix settings 2302.10663 app. A.

# Single-image textual inversion  [arxiv:2302.10663 §3.2] [app. A]
# Inputs: image (RGB, 512 px), init_emb (embedding of a starting word)
# Output: learned embedding for the new token <e>

def textual_inversion(image, init_emb, steps=3000):
    # 1. Only the new token's embedding is trainable; the rest is frozen
    emb = init_emb.clone().requires_grad_(True)
    for step in range(steps):
        # 2. Heavy random augmentation of the one image  [app. A]
        view = augment(image)
        # 3. Diffusion denoising loss with prompt "an image of a <e>"
        loss = diffusion_loss(view, prompt_with(emb))  # [eq. 3]
        # 4. Adam-type update, batch 16, lr 5e-4, weight decay 1e-2
        emb = adam_step(emb, loss, lr=5e-4, weight_decay=1e-2)
    return emb

Where the variants differ: the initial embedding can be random, hand-picked or chosen with CLIP, and the augmentation strength is a free choice within the listed ranges. The paper does not report how these choices change the result.

Contradictions

The paper and its released code differ on several settings. Both sides are cited here. The paper's stated values are the ones this page reports above.

• Optimiser. The paper says Adam with weight decay 1e-2 [arxiv:2302.10663 app. A]. The released code uses AdamW with the same decay, betas (0.9, 0.999) and epsilon 1e-8 textual_inversion.py.
• Effective batch and learning rate. The paper gives batch 16 and learning rate 5e-4 2302.10663 app. A. The released README command uses batch 1 with 4 accumulation steps and a learning-rate scaling flag, which gives an effective batch of 4 and a learning rate of 2e-3 [README.md#L123-L133] textual_inversion.py.
• Augmentation. The paper gives a random resized crop with scale between 0.70 and 1.30 2302.10663 app. A. The released code uses a scale of 0.85 to 1.15 with extra resize and crop steps textual_inversion.py.
• Prompt templates. The paper uses one prompt form, "an image of a " 2302.10663 §3.2. The released code samples from a set of ImageNet-style templates, such as "a photo of a {}" textual_inversion.py.

None of these changes is resolved here. For a faithful reimplementation, pick the paper values and say so; for a working system, the released code values are the ones the authors ran.

Papers

• RealFusion: 360° Reconstruction of Any Object from a Single Image: single-image textual inversion for a frozen Stable Diffusion, feeding a score distillation loss on random views of an Instant-NGP model.