---
kind: concept
slug: text-to-image-personalization
title: Text-to-image personalization
summary: >-
  Teaching a pretrained text-to-image diffusion model a new subject, such as a pet, an object, a person or a style, from a few photos or one image. Covers cross-attention key and value fine-tuning, joint and merged multi-concept training, pre-trained subject encoders, tuning-free multi-person generation, single-image editing, and training-free style sharing. Six compiled cards.
tags: [diffusion, personalization, text-to-image, fine-tuning, image-editing, style-consistency, multi-subject]
papers: ["2212.04488", "2212.04489", "2304.06025", "2305.10431", "2305.14720", "2312.02133"]
related: [attention-mechanism, classifier-free-guidance, ddim-inversion, diffusion-models, latent-diffusion, model-merging, parameter-efficient-fine-tuning, self-attention, text-guided-image-editing, vision-language-pretraining]
status: draft
updated_at: 2026-10-31
---

## Overview

Personalization means making a pretrained text-to-image model draw a concept
it has never seen, given a handful of photos of it. Naive full fine-tuning
does this, but it is slow, stores a full model copy per concept, and makes
the model forget how to draw related general words. Training on a rare
concept can make the category word ("dog") produce the new concept instead
[arxiv:2212.04488 §1] [arxiv:2212.04488 §3.1]. Inversion-based methods such as
Textual Inversion and DreamBooth need hundreds to thousands of steps for each
new subject [arxiv:2305.14720 §1].

The shared idea across the variants is to change as little as possible:

- Freeze most of the network and train only the weights that carry text
  information into the image model [arxiv:2212.04488 §3.1].
- Give the concept a word-like token and attach it to the training captions.
  The tuning variant learns the token [arxiv:2212.04488 §3.1]; the encoder
  variant produces the subject tokens from a photo with a pre-trained encoder
  [arxiv:2305.14720 §3.2]. The multi-person variant adds a photo embedding to
  each person's text token instead [arxiv:2305.10431 §4.1].
- Keep the general category usable, either by mixing in real images with
  ordinary captions [arxiv:2212.04488 §3.1] [app. E], or by pre-training on
  synthetic pairs so the model learns to rely on the subject
  [arxiv:2305.14720 §3.2].

The later variants depart from this. SINE and DreamPose tune the whole network
on one image, and StyleAligned trains nothing.

Five families are covered. Tuning methods optimize weights for each concept
[arxiv:2212.04488]. The encoder method predicts a subject embedding from the
photo and can work with no per-subject training [arxiv:2305.14720 §3.3].
The multi-person method takes one photo of each of several people, with no
per-person training, and is limited to persons [arxiv:2305.10431 §5.1]
[arxiv:2305.10431 §6]. Per-image tuning, in two cards, targets a single
picture: editing it with a prompt [arxiv:2212.04489 §3.2], or animating a
person along a pose sequence [arxiv:2304.06025 §4.3]. The last family shares
attention across a batch so that a set has one style, with no training
[arxiv:2312.02133 §3.2]. Most cards are written against Stable Diffusion; the
multi-person card uses v1-5 [arxiv:2305.10431 §5.1], and the style card uses
SDXL. The other methods they compare against, such as DreamBooth and Textual
Inversion, have no cards here and appear only as baselines.

## Variants

### Single concept: cross-attention key and value fine-tuning

Only the key and value projections of every cross-attention layer of the
U-Net are updated, together with the embedding of one new modifier token. The
authors measured that cross-attention changes most during fine-tuning while
holding a small share of the parameters, and text enters the image network
only through these two projections [arxiv:2212.04488 §3.1] [fig. 3] [fig. 4].
Each training batch is half target images and half real regularization
images, with a new token in the target caption [arxiv:2212.04488 app. E]. The
reported recipe is 250 steps at batch size 8 with a learning rate of
$8\times10^{-5}$ [arxiv:2212.04488 §3.2] [arxiv:2212.04488 app. E]. On 2 A100
GPUs training takes about 6 minutes, and the stored result is about 75MB per
concept [arxiv:2212.04488 §4.1]. The card is
[Multi-Concept Customization of Text-to-Image Diffusion](/papers/2212.04488).

Author-reported results on ten datasets [arxiv:2212.04488 tab. 1]:

| Method | Text-alignment | Image-alignment | KID (×10³) |
|---|---|---|---|
| Ours (single concept) | 0.795 | 0.775 | 20.96 |
| DreamBooth (third-party implementation) | 0.781 | 0.776 | 32.53 |
| Textual Inversion | 0.670 | 0.827 | 22.27 |

Fine-tuning all U-Net weights with the same recipe gives 0.795, 0.748 and
19.27 on the same table, which is the reference point for the claim that
restricting the update costs little [arxiv:2212.04488 tab. 1].

### Joint training for several concepts

Each concept gets its own modifier token. The training sets of all concepts
are combined, and the key and value projections and all tokens are trained at
once. The paper reports 500 steps for two concepts [arxiv:2212.04488 §3.2].
This variant has the best multi-concept text-alignment in the authors' table,
0.801 against 0.800 for the closed-form merge
[arxiv:2212.04488 tab. 1]. In a human preference study the joint model was
preferred over DreamBooth on text and image alignment at 56.39% and 61.80%
[arxiv:2212.04488 tab. 2].

### Closed-form merge of separately trained concepts

If each concept already has its own fine-tuned key and value matrices, they
can be combined without any further training. For each cross-attention layer
and each of key and value, the merged matrix is the one closest to the
pretrained matrix on regularization text features, subject to reproducing each
concept's fine-tuned output on its own words [arxiv:2212.04488 §3.2]
[arxiv:2212.04488 eq. 4] [arxiv:2212.04488 eq. 5]. The solve takes about 2
seconds once the single-concept models exist [arxiv:2212.04488 §3.2]. Sequential
training, which adds one concept after another, forgets the earlier concept
[arxiv:2212.04488 §4.2]; the closed-form merge is the authors' alternative
that does not need joint data.

The authors' table gives 0.800 and 0.695 for the merge "Optimization" row and
0.797 and 0.700 for "Sequential" [arxiv:2212.04488 tab. 1].

### Regularization and augmentation choices

Real regularization images are the main protection against drift. The
ablation without them, run for half the iterations, gives a KID of 32.64
against 20.96 with them [arxiv:2212.04488 tab. 3]. Using generated images
instead gives 34.70 [arxiv:2212.04488 tab. 3]. Random resizing of the target
images, with a matching size or zoom word added to the caption and the loss
restricted to the valid image area, lowers text-alignment slightly (0.795
against 0.800) and raises image-alignment (0.775 against 0.736), with KID about
the same (20.96 against 20.67) [arxiv:2212.04488 tab. 3]
[arxiv:2212.04488 app. E].

The starting point of the new token also matters, and the trade-off is not
uniform. Starting from a rare token and optimizing it gives the highest
image-alignment (0.774) but the highest KID (22.43) of the three variants
tested; a random start gives 0.765 and 22.06, and leaving the token
unoptimized gives the best text-alignment (0.816) and the lowest KID (21.86)
with the lowest image-alignment (0.733) [arxiv:2212.04488 app. C]
[arxiv:2212.04488 tab. 5].

### Compressed per-concept storage

The difference between fine-tuned and pretrained key and value matrices has
quickly decaying singular values, so it can be stored as a truncated SVD. The
authors report that keeping the top 60% of the singular value mass gives about
5× smaller storage with similar metrics, while the top 1% lowers
image-alignment [arxiv:2212.04488 app. C] [fig. 14] [fig. 15]. Training with a
low-rank constraint from the start gave worse results [arxiv:2212.04488 app. C].

### Pre-trained subject encoder, zero-shot or short fine-tune

This family does not optimize a token per concept. A frozen image encoder and
a trainable multimodal query module (BLIP-2's Q-Former, with 16 queries) turn
the photo and its category word into subject tokens. A small projection maps
them into the text embedding space, and they are added to the prompt
[arxiv:2305.14720 §3.1] [arxiv:2305.14720 §3.2]. The base model is Stable
Diffusion v1-5 [arxiv:2305.14720 §4.1]. The card is
[BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing](/papers/2305.14720).

Zero-shot use needs one photo and no training. The authors report DINO 0.594
on DreamBench, against 0.668 for DreamBooth and 0.500 for DreamBooth at 300
steps. DreamBooth at 100 steps reaches only 0.396 in the same table
[arxiv:2305.14720 tab. 1]. A short fine-tune averages the subject embeddings
of the few photos, then trains the U-Net with the text prompt. It reaches DINO
0.670 [arxiv:2305.14720 tab. 1]. The paper gives 40 to 120 steps per subject,
with a mean of 76 [arxiv:2305.14720 §3.3] [arxiv:2305.14720 app. A.5]. Its
per-subject table lists one subject at 20 steps [arxiv:2305.14720 tab. 3], so
plan for 20 to 120.

The multimodal encoder matters for the result. Removing its pre-training drops
zero-shot DINO from 0.566 to 0.521 at 250K pre-training steps
[arxiv:2305.14720 tab. 2]. The subject pre-training itself uses 500K steps on
16 A100 40GB GPUs, about six days [arxiv:2305.14720 §4.1]. The released code
includes only the fine-tuning configs, so a team cannot reproduce this stage
from the repository [arxiv:2305.14720 §4.1]; see the card's Paper vs code
section.

The authors also show the same model with a pre-trained ControlNet attached to
the U-Net, without further training, and text-driven editing through masks from
cross-attention maps [arxiv:2305.14720 §3.3] [arxiv:2305.14720 fig. 4].
Person-related subjects were removed from the pre-training data on purpose,
and the authors say this weakens person-as-subject control
[arxiv:2305.14720 app. A.1] [arxiv:2305.14720 app. A.4].

### Several people, one photo each: encoder embeddings with localized attention

Each person in the prompt has one token. That token's text embedding is
replaced by an MLP applied to the text embedding and a CLIP image embedding of
the person's photo. The other tokens keep their embeddings [arxiv:2305.10431
§4.1] [arxiv:2305.10431 eq. 2]. Subject crops have their background replaced
with random noise during training, so the model does not learn the background.
Natural photos can be used at test time without a segmentation step
[arxiv:2305.10431 §4.1].

Training adds a localization loss on the cross-attention map of each subject
token. It pushes attention inside the person's mask above attention outside it,
on the five middle U-Net blocks, with weight λ = 0.001 [arxiv:2305.10431 §4.2]
[arxiv:2305.10431 eq. 3] [arxiv:2305.10431 eq. 4]. Masks and this loss are used
only in training [arxiv:2305.10431 §4.2]. Without the loss, the authors report
identity preservation of 37.66% against 43.11% and prompt consistency of 25.03%
against 24.25% [arxiv:2305.10431 tab. 3]. The paper does not name the
evaluation set for that ablation.

Delayed subject conditioning limits the damage from copying the photo. While
the timestep is above α·T, the plain text conditioning is used; after that, the
subject-augmented conditioning is used [arxiv:2305.10431 §4.3]
[arxiv:2305.10431 eq. 5]. The authors suggest α between 0.6 and 0.8, and a
higher α raises identity preservation at the cost of editability
[arxiv:2305.10431 §4.3] [arxiv:2305.10431 fig. 5].

Training freezes the text encoder and trains the U-Net, the MLP and the last
two blocks of the image encoder, for 150,000 steps at batch size 128
[arxiv:2305.10431 §5.1]. The training pairs are built from an FFHQ-based set,
with captions and segmentation from models the paper names but the repository
does not contain [arxiv:2305.10431 §5.1].

The authors report results on 15 CelebA subjects, one photo each, against
Custom Diffusion ([/papers/2212.04488](/papers/2212.04488)), which uses five
photos. Identity preservation is 51.41% against 43.37%, and the total time is 2
s against 789 s [arxiv:2305.10431 tab. 1] [arxiv:2305.10431 §5.2]. On 105
pairs of those subjects, identity is 43.11% against 5.37% for Custom Diffusion
and 13.52% for Textual Inversion. Custom Diffusion has the higher prompt
consistency, 25.84% against 24.25% [arxiv:2305.10431 tab. 2] [arxiv:2305.10431
§5.3]. These are the authors' own tests, not independent ones. The paper limits
itself to at most three subjects in practice [arxiv:2305.10431 §6].


SINE fine-tunes Stable Diffusion on one picture, with the fine-tuning prompt
"a photo/painting of a [*] [class noun]" [arxiv:2212.04489 §3.1]. Training
uses random crops of the picture, each with a positional code, so the result
can be sampled at sizes other than the training size [arxiv:2212.04489 §3.3].
At sampling time, for the early steps (t > K, with K = 400), the prediction
mixes the fine-tuned and pretrained models with weight v = 0.7. Later steps use
the pretrained model alone [arxiv:2212.04489 §4]. The reported schedule is 800
steps without patches and 10,000 with them, at batch size 1 and base learning
rate 1e-6 [arxiv:2212.04489 §4]. The paper gives no numeric headline. Its
evidence is qualitative images and parameter sweeps, whose values are plotted
but not given in the text [arxiv:2212.04489 §4.3] [arxiv:2212.04489 fig. 8].
The card is [SINE: SINgle Image Editing with Text-to-Image Diffusion Models](/papers/2212.04489).

### Single image, video: per-subject tuning for pose-guided animation

DreamPose conditions a Stable Diffusion UNet on CLIP and VAE tokens of a photo,
and concatenates the pose maps to the noisy latent [arxiv:2304.06025 §4.2].
For each subject it tunes the UNet, then the VAE decoder, with random-crop
augmentation, because training on one frame and pose pair otherwise produces
texture sticking [arxiv:2304.06025 §4.3]. The step counts are 500 for the UNet
and 1500 for the decoder [arxiv:2304.06025 supp. §1]. Each frame is sampled
alone under two separate guidance weights, one for the image and one for the
pose [arxiv:2304.06025 §4.4] [arxiv:2304.06025 eq. 4]. The authors report
about 10 minutes of UNet tuning, 20 minutes of decoder tuning and 18 seconds per
rendered frame, without naming the hardware [arxiv:2304.06025 §6]. On UBC Fashion
they report FID 13.04, against 22.87 for TPSMM, the best baseline on that metric
[arxiv:2304.06025 tab. 1]. The card is
[DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion](/papers/2304.06025).

### Training-free: shared self-attention for one style

StyleAligned uses a pretrained SDXL model with no training [arxiv:2312.02133 §4].
In each self-attention layer, every image in a batch attends to its own keys
and values and to those of the first image, the reference [arxiv:2312.02133
§3.2]. The queries and keys of the other images are first normalised with AdaIN
to the reference's per-channel statistics. Values are not normalised
[arxiv:2312.02133 §3.2]. A real picture can serve as the reference through DDIM
inversion, which the authors say can fail [arxiv:2312.02133 §4.3]. Sharing can
be limited to a fraction of the layers, which gives more varied sets
[arxiv:2312.02133 §4.3]. On 100 prompts, the authors report a DINO set
consistency of 0.51 and a CLIP text alignment of 0.287. DreamBooth-LoRA, which
is trained per style, scores 0.537 and 0.276 [arxiv:2312.02133 tab. 2]. The
card is [Style Aligned Image Generation via Shared Attention](/papers/2312.02133).

## When to use which

| Situation | Use | Why |
|---|---|---|
| One concept, a small file per concept is needed | Single-concept key and value fine-tuning | About 75MB against about 3GB for a full DreamBooth copy, in minutes [arxiv:2212.04488 §4.1] |
| Several concepts that must appear together, and training time is available | Joint training with one token per concept | Best multi-concept text-alignment in the authors' table [arxiv:2212.04488 tab. 1] |
| Concepts were already trained separately and a quick combination is needed | Closed-form merge | About 2 s, with a small quality loss against joint training [arxiv:2212.04488 §3.2] [arxiv:2212.04488 tab. 1] |
| Many concepts or sequential additions | Check first | Results cover pairs, and three concepts only for some combinations [arxiv:2212.04488 fig. 16] |
| Training data is only a few images and general prompts must keep working | Keep real regularization images | Without them the KID rises sharply [arxiv:2212.04488 tab. 3] |
| One subject from a single photo, and no training run is acceptable | Encoder method, zero-shot | No per-subject training; DINO 0.594 in the authors' table, below the fine-tuned 0.670 [arxiv:2305.14720 tab. 1] |
| Subject fidelity matters and a few photos plus a short fine-tune are acceptable | Encoder method, fine-tuned | DINO 0.670 with 20 to 120 steps [arxiv:2305.14720 tab. 1] [arxiv:2305.14720 tab. 3] |
| Pose or structure control on top of the subject | Encoder method with ControlNet | Attachable without further training, per the authors [arxiv:2305.14720 §3.3] [arxiv:2305.14720 fig. 4] |
| Several people in one image, one photo each, and no training run per person | Multi-person encoder method | Identity 43.11% against 5.37% for Custom Diffusion on the authors' 105 pairs, with lower prompt consistency (24.25% against 25.84%) [arxiv:2305.10431 tab. 2]; persons only [arxiv:2305.10431 §5.1] |
| Several subjects that are objects or animals | Joint training or closed-form merge | The multi-person card's training and scope cover persons only [arxiv:2305.10431 §5.1] [arxiv:2305.10431 §6]; for the tuning methods' multi-concept results, see [arxiv:2212.04488 tab. 1] |
| One picture to edit with a text prompt, keeping its content, and per-image tuning is acceptable | Single-image fine-tune with guidance mixing | No numeric headline; the paper's evidence is qualitative, so test on your own images [arxiv:2212.04489 §4.2] |
| One person's photo to animate along a pose sequence, with per-subject tuning for each person | Per-subject tuning for pose-guided video | About 30 minutes of tuning per subject, then 18 s per frame [arxiv:2304.06025 §6]; FID 13.04 in the authors' table [arxiv:2304.06025 tab. 1] |
| A set of images in one style, with no training, on SDXL | Shared self-attention | Set consistency 0.51 against 0.537 for the per-style DreamBooth-LoRA, which is more consistent but needs training [arxiv:2312.02133 tab. 2] |
| Base model must be reproduced from scratch | Neither, from this wiki's cards | The encoder stage-2 code is not released [arxiv:2305.14720 §4.1]; the tuning method's code is non-commercial (see its card) |
| Model access is limited to an API without weight updates | Not covered | Every variant needs the U-Net internals, as weights or as attention layers at sampling time [arxiv:2212.04488 §3.1] [arxiv:2305.14720 §3.2] [arxiv:2312.02133 §3.2] |

For commercial code reuse, the encoder method's repository is BSD-3-Clause
([/papers/2305.14720](/papers/2305.14720)), while the tuning method's code
carries a non-commercial research license ([/papers/2212.04488](/papers/2212.04488)).
The code of the per-image cards is MIT ([/papers/2212.04489](/papers/2212.04489),
[/papers/2304.06025](/papers/2304.06025)), and StyleAligned's is Apache-2.0
([/papers/2312.02133](/papers/2312.02133)). The multi-person card's code is
MIT-licensed ([/papers/2305.10431](/papers/2305.10431)), but the paper does not
state the licenses of the Stable Diffusion weights, the FFHQ-based training
data or the released checkpoint [arxiv:2305.10431 §5.1]; check them before
commercial use. The DreamPose card records that its
checkpoints have no stated license and that its training datasets are third
party, so the code license does not cover them. Check each card's license
before copying anything.

Choose the step count with the trade-off in mind: text-alignment falls and
image-alignment rises as training continues [arxiv:2212.04488 app. D]
[fig. 20]. For the encoder method, the authors picked each checkpoint by hand
against validation prompts, and that selection is not counted in the 40 to 120
steps [arxiv:2305.14720 app. A.5].

## Implementation sketch

Language-agnostic outlines of the single-concept tuning variant, the encoder
variant's zero-shot path, the multi-person conditioning and shared attention.
Variant differences are noted after each.
Details are in the cards.

```pseudocode
# Single-concept personalization  [§3.1] [§3.2] [app. E]
# Input: 4 to ~135 target images, a category word, a pretrained model,
#        ~200 real images with captions close to the target caption
# Output: updated key/value projections plus one new token embedding

def personalize(target_images, category, model, reg_set):
    # new modifier token, starting from a rarely used token embedding  [app. C]
    v_star = add_token(model.tokenizer, rare_init=True)
    # trainable: key and value projections of every cross-attention layer,
    # and the new token embedding  [§3.1]
    params = cross_attention_kv(model.unet) + [embedding(v_star)]
    # step count and rate from [§3.2] and [app. E]
    opt = make_optimizer(params, lr=8e-5)
    for step in range(250):
        # half target, half real regularization samples  [app. E]
        batch = sample_balanced(target_images, reg_set, batch_size=8)
        # target samples get the new token and a random size or zoom phrase
        # with a loss mask  [app. E]
        loss = denoising_loss(model, batch)   # [eq. 2] with masks
        loss.backward()
        opt.step()
        opt.zero_grad()
    return params
```

Variant differences: joint training adds one token per concept and merges
their batches [§3.2]. The closed-form merge replaces the loop with a single
least-squares solve per layer and per key or value matrix [eq. 4] [eq. 5].

Encoder method, zero-shot path. No training happens here; the photo goes
straight into the prompt conditioning [§3.1] [§3.2] [§3.3].

```pseudocode
# Pre-trained subject encoder, zero-shot path  [§3.1] [§3.2] [app. A.5]
# Input: one subject photo, its category word, a prompt
# Output: an image that carries the subject, with no per-subject training

def generate_zero_shot(subject_img, category, prompt, model):
    # 16 subject tokens from the multimodal query module  [§3.1]
    queries = multimodal_encoder(subject_img, category)
    # project them into the text embedding space  [§3.2]
    subject_tokens = project(queries)
    # insert position is not stated in the paper  [§3.2]
    embeds = splice(clip_text(prompt), subject_tokens)
    # PNDM sampler with classifier-free guidance 7.5  [app. A.5]
    return sample(model.unet, embeds, scale=7.5, sampler="PNDM")
```

The fine-tuned path replaces the direct call with the mean of the photo
embeddings, then trains the U-Net for 40 to 120 steps [§3.3] [app. A.5].

Multi-person path, one photo per person. Only the prompt embeddings change,
and the early sampling steps use the plain text [§4.1] [§4.3].

```pseudocode
# Subject-augmented conditioning for several people  [§4.1] [eq. 2]
# Input: prompt tokens (n,), one photo per person (m, 3, H, W), token index
#        per person (m,)
# Output: conditioning embeddings (n, d)

def augment_prompt(prompt_tokens, photos, person_token_index):
    # Encode the prompt with the text encoder  [§3]
    text_embeds = text_encoder(prompt_tokens)
    # Encode each photo with the image encoder  [§4.1]
    photo_embeds = image_encoder(photos)
    for j, i in enumerate(person_token_index):
        # Only the person's token is replaced, by an MLP over both  [eq. 2]
        text_embeds[i] = mlp(concat(text_embeds[i], photo_embeds[j]))
    return text_embeds
```

Variant differences: the text-only conditioning is used while the timestep is
above α·T, then this augmented conditioning [§4.3] [eq. 5]. The released MLP is
deeper than eq. 2 (see the card's Paper vs code section).

Training-free shared attention, for a set of images in one style. Only the
attention call changes; no weights are trained [§3.2].

```pseudocode
# Shared self-attention for one layer; index 0 is the reference  [§3.2]
# queries, keys, values: (N, m, d); output: (N, m, d)

def shared_self_attention(q, k, v):
    # Match the targets' queries and keys to the reference statistics  [§3.2]
    q, k = adain_to_reference(q), adain_to_reference(k)
    out = []
    for i in range(N):
        # Each image attends to its own tokens and the reference's  [§3.2]
        keys = concat(k[i], k[0])
        vals = concat(v[i], v[0])
        out.append(softmax(q[i] @ keys.T / sqrt(d)) @ vals)
    return stack(out)
```

## Contradictions

- **Training steps and paper versus released code.** The paper reports 250
  steps for one concept and 500 for two [arxiv:2212.04488 §3.2]. The card's
  Paper vs code section records that the released configuration files use
  different counts, so reproduce from the paper numbers and check the card
  before copying any setting.
- **Retrieval threshold.** The paper selects regularization captions with a
  CLIP text similarity above 0.85 [arxiv:2212.04488 §3.1]. The released
  retrieval step does not apply that threshold (see the card).
- **Regularization source in the robustness study.** Real retrieved images give
  the better KID on the main dataset set [arxiv:2212.04488 tab. 3]. The
  CustomConcept101 comparison uses generated regularization images for both
  this method and DreamBooth [arxiv:2212.04488 app. A] [arxiv:2212.04488 tab. 4].
  Those numbers should not be compared directly with the main table.
- **Table 4 bold marks.** In the CustomConcept101 multi-concept rows, the
  printed bold marks the joint model for text-alignment, but the value printed
  for the closed-form merge is higher (0.763 against 0.757)
  [arxiv:2212.04488 tab. 4].
- **Baseline implementation.** The DreamBooth baseline is a third-party
  implementation with its own learning rate and step settings
  [arxiv:2212.04488 app. E]. Head-to-head numbers reflect those choices as
  well as the method.
- **Prompt phrase placement.** The paper describes the size and zoom phrases as
  appended to the prompt [arxiv:2212.04488 §3.2]. The released code prepends
  them (see the card).
- **Fine-tune learning rate, encoder method.** The text gives 5e-5 for the
  fine-tune [arxiv:2305.14720 §3.3]; the appendix and the released config give
  5e-6 [arxiv:2305.14720 app. A.5]. The card suggests 5e-6 unless tested
  otherwise.
- **Inference steps, encoder method.** The paper uses PNDM with 100 steps
  [arxiv:2305.14720 app. A.5]. The released code defaults to 50 (see the card).
- **Subject-prompt dropout, encoder method.** The paper describes dropping the
  subject prompt in 15% of pre-training steps [arxiv:2305.14720 §3.2]. The
  released code has no such step, but it also omits the pre-training stage
  (see the card).
- **Zero-shot against fine-tuned Textual Inversion, encoder method.** The text
  says zero-shot beats fine-tuned Textual Inversion [arxiv:2305.14720 §4.2]. The
  table agrees on DINO and CLIP-T, but on CLIP-I it gives 0.779 against 0.780
  [arxiv:2305.14720 tab. 1].
- **DreamBooth numbers across the two papers.** The tuning card's table reports
  a third-party DreamBooth with KID and text/image alignment
  [arxiv:2212.04488 tab. 1]. The encoder card's table reports DINO, CLIP-I and
  CLIP-T [arxiv:2305.14720 tab. 1]. The metric sets and baselines differ, so
  the numbers should not be compared across the two cards.

- **SINE guidance window.** The paper applies the fine-tuned term for t > K
  [arxiv:2212.04489 §3.2]. The sampler's default arguments apply it for
  small t instead, while the notebook and README use the paper's window (see
  the card's Paper vs code section). Copy the window from the notebook.
- **DreamPose pose window.** The paper conditions on five consecutive poses
  [arxiv:2304.06025 §4.2.2]. The released training code fills the window with
  one pose repeated five times (see the card). A model trained from that code
  does not see the input the paper describes.
- **DreamPose frame size.** The paper trains at 512 × 512 [arxiv:2304.06025
  supp. §1] and evaluates at 256 px [arxiv:2304.06025 §5.1.1]. The released
  code hard-codes 640 × 512 (see the card).
- **StyleAligned reference keys.** The paper writes the targets' keys as
  [K_r; K̂_t] [arxiv:2312.02133 §3.2]. The released code also appends the
  reference's keys to its own attention, which the paper does not describe
  (see the card). The reference's output is unaffected by this, per the card's
  reading, but the code was not run.
- **Delayed-conditioning switch, multi-person method.** Eq. 5 switches on the
  timestep [arxiv:2305.10431 eq. 5]. The released sampler switches on the step
  index, and its shipped script starts subject conditioning at an absolute step,
  so the number of text-only steps differs from α·T (see the card's Paper vs
  code section). Count the steps from the code, not the equation.
- **Speed-up claims, multi-person method.** The introduction gives 300× to
  2,500× [arxiv:2305.10431 §1] and the experiments 300× to 1,200×
  [arxiv:2305.10431 §5.2]. Ratios computed from the authors' tables are 394× to
  1,250× for one subject and 394× to 2,499× for several [arxiv:2305.10431 tab. 1]
  [arxiv:2305.10431 tab. 2]. The paper's own ranges match neither exactly.
- **Identity metric, multi-person method.** The paper uses FaceNet similarity of
  detected faces [arxiv:2305.10431 §5.1]. The released evaluation uses a
  VGGFace2-trained model instead (see the card). Its identity scores are face
  scores and are not directly comparable to the DINO, KID or CLIP numbers in the
  other cards.
- **Baseline coverage, multi-person method.** Section 5.3 cites DreamBooth among
  the optimization baselines, but the multi-subject table has no DreamBooth row
  [arxiv:2305.10431 §5.3] [arxiv:2305.10431 tab. 2].
- **Conditioning MLP, multi-person method.** Eq. 2 is a single MLP. The released
  code uses residual blocks with normalization, so a literal reading of eq. 2
  will not match the released checkpoint [arxiv:2305.10431 §4.1]
  [arxiv:2305.10431 eq. 2].

## Papers

- [Multi-Concept Customization of Text-to-Image Diffusion](/papers/2212.04488): cross-attention key and value fine-tuning with a new token, joint training and a closed-form merge for several concepts. The authors' code is non-commercially licensed; reimplement from the paper for commercial use.
- [BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing](/papers/2305.14720): a pre-trained multimodal subject encoder that gives zero-shot subject generation and fine-tunes in tens of steps. The released code covers fine-tuning only, under BSD-3-Clause.
- [FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention](/papers/2305.10431): one reference photo per person, embedded with CLIP and inserted into the text tokens, with a cross-attention localization loss in training and delayed subject conditioning at sampling. Persons only; the authors' results are on their own test sets. The code is MIT-licensed.
- [SINE: SINgle Image Editing with Text-to-Image Diffusion Models](/papers/2212.04489): fine-tunes Stable Diffusion on one picture with random patch crops and a positional code, then edits by mixing the fine-tuned and pretrained predictions in early steps. Results are qualitative. The code is MIT-licensed.
- [DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion](/papers/2304.06025): per-subject tuning of a pose-conditioned Stable Diffusion for pose-guided fashion video, with separate image and pose guidance weights. The code is MIT-licensed.
- [Style Aligned Image Generation via Shared Attention](/papers/2312.02133): training-free shared self-attention for a set of SDXL images in one style. The code is Apache-2.0.