---
kind: concept
slug: textual-inversion
title: Textual inversion
summary: >-
  Learning one new token embedding for a frozen text-to-image diffusion model, so that a made-up word stands for one object or concept. Covers the single-image variant from RealFusion: augmented copies of one image, batch, learning rate, token initialisation, and where the released code departs from the paper. One compiled card so far.
tags: [diffusion, personalization, text-embedding, 3d-reconstruction]
papers: ["2302.10663"]
related: [diffusion-models, text-to-image-personalization, score-distillation-sampling, image-to-3d-generation, neural-radiance-field, soft-prompt-tuning, classifier-free-guidance, text-embedding]
status: draft
updated_at: 2026-11-01
---

## Overview

Textual inversion teaches a text-to-image diffusion model a new word for one
concept. The only parameter trained is the embedding vector of a new token;
the other embeddings and all model weights stay frozen. The learning signal
is the diffusion denoising loss on the images that show the concept
[arxiv:2302.10663 §3.2] [arxiv:2302.10663 app. A].

The one compiled card covers a single use of the idea, in which one image
replaces a set of photos. Other variants have no card yet, so this page
cannot compare them.

## Variants

### Single-image textual inversion (RealFusion)

The input is one image of an object, with the background removed. The paper
cannot get other views of the object, so it builds a small training set from
random augmentations of that image. It then learns one token embedding with
the diffusion loss, using the prompt "an image of a <e>" [arxiv:2302.10663
§3.2]. The augmentations are a rotation of up to 10°, a random resized crop,
colour jitter, occasional greyscale and blur, and a horizontal flip. The
paper gives these as a listing in its appendix [arxiv:2302.10663 app. A].

The paper's stated settings are 3000 steps, batch 16, learning rate 5e-4,
weight decay 1e-2 with Adam, and 512 px images [arxiv:2302.10663 app. A]. The
token can start from a random vector, from a word chosen by hand, or from the
best-matching noun found with CLIP over WordNet. The paper used the manual
start for its main figures and CLIP for its supplement [arxiv:2302.10663
app. A]. The learned token is then used inside the prompt of a score
distillation loss on random views of a 3D model. That loss is the other half
of the method [arxiv:2302.10663 §3.2] and is covered on the
[Score distillation sampling](/concepts/score-distillation-sampling) page.
The card for the full method is [RealFusion: 360° Reconstruction of Any
Object from a Single Image](/papers/2302.10663).

Trade-off: the paper says this step is its core contribution [arxiv:2302.10663
§3.2]. The card records no ablation that switches this step off, so its share
of the reported results is not measured here.

### Other variants

The original textual inversion paper, [arXiv:2208.01618](https://arxiv.org/abs/2208.01618),
has no card yet, so nothing from it is used on this page.

## When to use which

| Situation | Use | Why |
|---|---|---|
| You have one image of the object and need a 3D model from it | Single-image textual inversion, then score distillation | The paper builds the training set from augmentations of the one image, and its evaluation makes one reconstruction per input image [arxiv:2302.10663 §3.2] [arxiv:2302.10663 §4.2] |
| You have several photos of the subject and want it in 2D prompts | Not covered | No card for a multi-image variant yet |
| You need the frozen model to stay as it is | Textual inversion | Only the embedding is trained; the weights are not changed [arxiv:2302.10663 §3.2] |
| You need the training cost for a new concept | Not stated | The paper gives the step count and batch size but no wall-clock time for this stage. It reports about 45 minutes on one V100 for the reconstruction alone [arxiv:2302.10663 app. A] |
| You must reproduce the paper's numbers | Follow the paper's settings and read the contradictions below | The released code uses different optimiser, augmentation and learning-rate scaling |

The paper's reported comparison is for the whole RealFusion pipeline against
Shelf-Supervised Mesh Prediction on 7 CO3D categories, not for this step on
its own. On mean F-score the full method reports 9.58 against 8.24 [arxiv:2302.10663
tab. 1]. Treat that as the result of the pipeline.

## Implementation sketch

Only the token embedding is trained. Everything else is frozen. The steps
below follow the paper's appendix settings [arxiv:2302.10663 app. A].

```pseudocode
# Single-image textual inversion  [arxiv:2302.10663 §3.2] [app. A]
# Inputs: image (RGB, 512 px), init_emb (embedding of a starting word)
# Output: learned embedding for the new token <e>

def textual_inversion(image, init_emb, steps=3000):
    # 1. Only the new token's embedding is trainable; the rest is frozen
    emb = init_emb.clone().requires_grad_(True)
    for step in range(steps):
        # 2. Heavy random augmentation of the one image  [app. A]
        view = augment(image)
        # 3. Diffusion denoising loss with prompt "an image of a <e>"
        loss = diffusion_loss(view, prompt_with(emb))  # [eq. 3]
        # 4. Adam-type update, batch 16, lr 5e-4, weight decay 1e-2
        emb = adam_step(emb, loss, lr=5e-4, weight_decay=1e-2)
    return emb
```

Where the variants differ: the initial embedding can be random, hand-picked
or chosen with CLIP, and the augmentation strength is a free choice within the
listed ranges. The paper does not report how these choices change the result.

## Contradictions

The paper and its released code differ on several settings. Both sides are
cited here. The paper's stated values are the ones this page reports above.

- **Optimiser.** The paper says Adam with weight decay 1e-2 [arxiv:2302.10663
  app. A]. The released code uses AdamW with the same decay, betas (0.9, 0.999)
  and epsilon 1e-8 [code: textual-inversion/textual_inversion.py#L262-L265].
- **Effective batch and learning rate.** The paper gives batch 16 and learning
  rate 5e-4 [arxiv:2302.10663 app. A]. The released README command uses batch 1
  with 4 accumulation steps and a learning-rate scaling flag, which gives an
  effective batch of 4 and a learning rate of 2e-3 [README.md#L123-L133]
  [code: textual-inversion/textual_inversion.py#L695-L698].
- **Augmentation.** The paper gives a random resized crop with scale between
  0.70 and 1.30 [arxiv:2302.10663 app. A]. The released code uses a scale of
  0.85 to 1.15 with extra resize and crop steps
  [code: textual-inversion/textual_inversion.py#L492-L505].
- **Prompt templates.** The paper uses one prompt form, "an image of a <e>"
  [arxiv:2302.10663 §3.2]. The released code samples from a set of
  ImageNet-style templates, such as "a photo of a {}"
  [code: textual-inversion/textual_inversion.py#L392-L393].

None of these changes is resolved here. For a faithful reimplementation, pick
the paper values and say so; for a working system, the released code values are
the ones the authors ran.

## Papers

- [RealFusion: 360° Reconstruction of Any Object from a Single Image](/papers/2302.10663):
  single-image textual inversion for a frozen Stable Diffusion, feeding a
  score distillation loss on random views of an Instant-NGP model.