Text-to-image personalization

Overview

Personalization means making a pretrained text-to-image model draw a concept it has never seen, given a handful of photos of it. Naive full fine-tuning does this, but it is slow, stores a full model copy per concept, and makes the model forget how to draw related general words. Training on a rare concept can make the category word ("dog") produce the new concept instead 2212.04488 §1 2212.04488 §3.1. Inversion-based methods such as Textual Inversion and DreamBooth need hundreds to thousands of steps for each new subject 2305.14720 §1.

The shared idea across the variants is to change as little as possible:

• Freeze most of the network and train only the weights that carry text information into the image model 2212.04488 §3.1.
• Give the concept a word-like token and attach it to the training captions. The tuning variant learns the token 2212.04488 §3.1; the encoder variant produces the subject tokens from a photo with a pre-trained encoder 2305.14720 §3.2. The multi-person variant adds a photo embedding to each person's text token instead 2305.10431 §4.1.
• Keep the general category usable, either by mixing in real images with ordinary captions 2212.04488 §3.1 app. E, or by pre-training on synthetic pairs so the model learns to rely on the subject 2305.14720 §3.2.

The later variants depart from this. SINE and DreamPose tune the whole network on one image, and StyleAligned trains nothing.

Five families are covered. Tuning methods optimize weights for each concept 2212.04488. The encoder method predicts a subject embedding from the photo and can work with no per-subject training 2305.14720 §3.3. The multi-person method takes one photo of each of several people, with no per-person training, and is limited to persons 2305.10431 §5.1 2305.10431 §6. Per-image tuning, in two cards, targets a single picture: editing it with a prompt 2212.04489 §3.2, or animating a person along a pose sequence 2304.06025 §4.3. The last family shares attention across a batch so that a set has one style, with no training 2312.02133 §3.2. Most cards are written against Stable Diffusion; the multi-person card uses v1-5 2305.10431 §5.1, and the style card uses SDXL. The other methods they compare against, such as DreamBooth and Textual Inversion, have no cards here and appear only as baselines.

Variants

Single concept: cross-attention key and value fine-tuning

Only the key and value projections of every cross-attention layer of the U-Net are updated, together with the embedding of one new modifier token. The authors measured that cross-attention changes most during fine-tuning while holding a small share of the parameters, and text enters the image network only through these two projections 2212.04488 §3.1 fig. 3 fig. 4. Each training batch is half target images and half real regularization images, with a new token in the target caption 2212.04488 app. E. The reported recipe is 250 steps at batch size 8 with a learning rate of 8\times10^{-5} 2212.04488 §3.2 2212.04488 app. E. On 2 A100 GPUs training takes about 6 minutes, and the stored result is about 75MB per concept 2212.04488 §4.1. The card is Multi-Concept Customization of Text-to-Image Diffusion.

Author-reported results on ten datasets 2212.04488 tab. 1:

Fine-tuning all U-Net weights with the same recipe gives 0.795, 0.748 and 19.27 on the same table, which is the reference point for the claim that restricting the update costs little 2212.04488 tab. 1.

Joint training for several concepts

Each concept gets its own modifier token. The training sets of all concepts are combined, and the key and value projections and all tokens are trained at once. The paper reports 500 steps for two concepts 2212.04488 §3.2. This variant has the best multi-concept text-alignment in the authors' table, 0.801 against 0.800 for the closed-form merge 2212.04488 tab. 1. In a human preference study the joint model was preferred over DreamBooth on text and image alignment at 56.39% and 61.80% 2212.04488 tab. 2.

Closed-form merge of separately trained concepts

If each concept already has its own fine-tuned key and value matrices, they can be combined without any further training. For each cross-attention layer and each of key and value, the merged matrix is the one closest to the pretrained matrix on regularization text features, subject to reproducing each concept's fine-tuned output on its own words 2212.04488 §3.2 2212.04488 eq. 4 2212.04488 eq. 5. The solve takes about 2 seconds once the single-concept models exist 2212.04488 §3.2. Sequential training, which adds one concept after another, forgets the earlier concept 2212.04488 §4.2; the closed-form merge is the authors' alternative that does not need joint data.

The authors' table gives 0.800 and 0.695 for the merge "Optimization" row and 0.797 and 0.700 for "Sequential" 2212.04488 tab. 1.

Regularization and augmentation choices

Real regularization images are the main protection against drift. The ablation without them, run for half the iterations, gives a KID of 32.64 against 20.96 with them 2212.04488 tab. 3. Using generated images instead gives 34.70 2212.04488 tab. 3. Random resizing of the target images, with a matching size or zoom word added to the caption and the loss restricted to the valid image area, lowers text-alignment slightly (0.795 against 0.800) and raises image-alignment (0.775 against 0.736), with KID about the same (20.96 against 20.67) 2212.04488 tab. 3 2212.04488 app. E.

The starting point of the new token also matters, and the trade-off is not uniform. Starting from a rare token and optimizing it gives the highest image-alignment (0.774) but the highest KID (22.43) of the three variants tested; a random start gives 0.765 and 22.06, and leaving the token unoptimized gives the best text-alignment (0.816) and the lowest KID (21.86) with the lowest image-alignment (0.733) 2212.04488 app. C 2212.04488 tab. 5.

Compressed per-concept storage

The difference between fine-tuned and pretrained key and value matrices has quickly decaying singular values, so it can be stored as a truncated SVD. The authors report that keeping the top 60% of the singular value mass gives about 5× smaller storage with similar metrics, while the top 1% lowers image-alignment 2212.04488 app. C fig. 14 fig. 15. Training with a low-rank constraint from the start gave worse results 2212.04488 app. C.

Pre-trained subject encoder, zero-shot or short fine-tune

This family does not optimize a token per concept. A frozen image encoder and a trainable multimodal query module (BLIP-2's Q-Former, with 16 queries) turn the photo and its category word into subject tokens. A small projection maps them into the text embedding space, and they are added to the prompt 2305.14720 §3.1 2305.14720 §3.2. The base model is Stable Diffusion v1-5 2305.14720 §4.1. The card is BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing.

Zero-shot use needs one photo and no training. The authors report DINO 0.594 on DreamBench, against 0.668 for DreamBooth and 0.500 for DreamBooth at 300 steps. DreamBooth at 100 steps reaches only 0.396 in the same table 2305.14720 tab. 1. A short fine-tune averages the subject embeddings of the few photos, then trains the U-Net with the text prompt. It reaches DINO 0.670 2305.14720 tab. 1. The paper gives 40 to 120 steps per subject, with a mean of 76 2305.14720 §3.3 2305.14720 app. A.5. Its per-subject table lists one subject at 20 steps 2305.14720 tab. 3, so plan for 20 to 120.

The multimodal encoder matters for the result. Removing its pre-training drops zero-shot DINO from 0.566 to 0.521 at 250K pre-training steps 2305.14720 tab. 2. The subject pre-training itself uses 500K steps on 16 A100 40GB GPUs, about six days 2305.14720 §4.1. The released code includes only the fine-tuning configs, so a team cannot reproduce this stage from the repository 2305.14720 §4.1; see the card's Paper vs code section.

The authors also show the same model with a pre-trained ControlNet attached to the U-Net, without further training, and text-driven editing through masks from cross-attention maps 2305.14720 §3.3 2305.14720 fig. 4. Person-related subjects were removed from the pre-training data on purpose, and the authors say this weakens person-as-subject control 2305.14720 app. A.1 2305.14720 app. A.4.

Several people, one photo each: encoder embeddings with localized attention

Each person in the prompt has one token. That token's text embedding is replaced by an MLP applied to the text embedding and a CLIP image embedding of the person's photo. The other tokens keep their embeddings [arxiv:2305.10431 §4.1] 2305.10431 eq. 2. Subject crops have their background replaced with random noise during training, so the model does not learn the background. Natural photos can be used at test time without a segmentation step 2305.10431 §4.1.

Training adds a localization loss on the cross-attention map of each subject token. It pushes attention inside the person's mask above attention outside it, on the five middle U-Net blocks, with weight λ = 0.001 2305.10431 §4.2 2305.10431 eq. 3 2305.10431 eq. 4. Masks and this loss are used only in training 2305.10431 §4.2. Without the loss, the authors report identity preservation of 37.66% against 43.11% and prompt consistency of 25.03% against 24.25% 2305.10431 tab. 3. The paper does not name the evaluation set for that ablation.

Delayed subject conditioning limits the damage from copying the photo. While the timestep is above α·T, the plain text conditioning is used; after that, the subject-augmented conditioning is used 2305.10431 §4.3 2305.10431 eq. 5. The authors suggest α between 0.6 and 0.8, and a higher α raises identity preservation at the cost of editability 2305.10431 §4.3 2305.10431 fig. 5.

Training freezes the text encoder and trains the U-Net, the MLP and the last two blocks of the image encoder, for 150,000 steps at batch size 128 2305.10431 §5.1. The training pairs are built from an FFHQ-based set, with captions and segmentation from models the paper names but the repository does not contain 2305.10431 §5.1.

The authors report results on 15 CelebA subjects, one photo each, against Custom Diffusion (/papers/2212.04488), which uses five photos. Identity preservation is 51.41% against 43.37%, and the total time is 2 s against 789 s 2305.10431 tab. 1 2305.10431 §5.2. On 105 pairs of those subjects, identity is 43.11% against 5.37% for Custom Diffusion and 13.52% for Textual Inversion. Custom Diffusion has the higher prompt consistency, 25.84% against 24.25% 2305.10431 tab. 2 [arxiv:2305.10431 §5.3]. These are the authors' own tests, not independent ones. The paper limits itself to at most three subjects in practice 2305.10431 §6.

SINE fine-tunes Stable Diffusion on one picture, with the fine-tuning prompt "a photo/painting of a [*] [class noun]" 2212.04489 §3.1. Training uses random crops of the picture, each with a positional code, so the result can be sampled at sizes other than the training size 2212.04489 §3.3. At sampling time, for the early steps (t > K, with K = 400), the prediction mixes the fine-tuned and pretrained models with weight v = 0.7. Later steps use the pretrained model alone 2212.04489 §4. The reported schedule is 800 steps without patches and 10,000 with them, at batch size 1 and base learning rate 1e-6 2212.04489 §4. The paper gives no numeric headline. Its evidence is qualitative images and parameter sweeps, whose values are plotted but not given in the text 2212.04489 §4.3 2212.04489 fig. 8. The card is SINE: SINgle Image Editing with Text-to-Image Diffusion Models.

Single image, video: per-subject tuning for pose-guided animation

DreamPose conditions a Stable Diffusion UNet on CLIP and VAE tokens of a photo, and concatenates the pose maps to the noisy latent 2304.06025 §4.2. For each subject it tunes the UNet, then the VAE decoder, with random-crop augmentation, because training on one frame and pose pair otherwise produces texture sticking 2304.06025 §4.3. The step counts are 500 for the UNet and 1500 for the decoder 2304.06025 supp. §1. Each frame is sampled alone under two separate guidance weights, one for the image and one for the pose 2304.06025 §4.4 2304.06025 eq. 4. The authors report about 10 minutes of UNet tuning, 20 minutes of decoder tuning and 18 seconds per rendered frame, without naming the hardware 2304.06025 §6. On UBC Fashion they report FID 13.04, against 22.87 for TPSMM, the best baseline on that metric 2304.06025 tab. 1. The card is DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion.

Training-free: shared self-attention for one style

StyleAligned uses a pretrained SDXL model with no training 2312.02133 §4. In each self-attention layer, every image in a batch attends to its own keys and values and to those of the first image, the reference [arxiv:2312.02133 §3.2]. The queries and keys of the other images are first normalised with AdaIN to the reference's per-channel statistics. Values are not normalised 2312.02133 §3.2. A real picture can serve as the reference through DDIM inversion, which the authors say can fail 2312.02133 §4.3. Sharing can be limited to a fraction of the layers, which gives more varied sets 2312.02133 §4.3. On 100 prompts, the authors report a DINO set consistency of 0.51 and a CLIP text alignment of 0.287. DreamBooth-LoRA, which is trained per style, scores 0.537 and 0.276 2312.02133 tab. 2. The card is Style Aligned Image Generation via Shared Attention.

When to use which

For commercial code reuse, the encoder method's repository is BSD-3-Clause (/papers/2305.14720), while the tuning method's code carries a non-commercial research license (/papers/2212.04488). The code of the per-image cards is MIT (/papers/2212.04489, /papers/2304.06025), and StyleAligned's is Apache-2.0 (/papers/2312.02133). The multi-person card's code is MIT-licensed (/papers/2305.10431), but the paper does not state the licenses of the Stable Diffusion weights, the FFHQ-based training data or the released checkpoint 2305.10431 §5.1; check them before commercial use. The DreamPose card records that its checkpoints have no stated license and that its training datasets are third party, so the code license does not cover them. Check each card's license before copying anything.

Choose the step count with the trade-off in mind: text-alignment falls and image-alignment rises as training continues 2212.04488 app. D fig. 20. For the encoder method, the authors picked each checkpoint by hand against validation prompts, and that selection is not counted in the 40 to 120 steps 2305.14720 app. A.5.

Implementation sketch

Language-agnostic outlines of the single-concept tuning variant, the encoder variant's zero-shot path, the multi-person conditioning and shared attention. Variant differences are noted after each. Details are in the cards.

# Single-concept personalization  [§3.1] [§3.2] [app. E]
# Input: 4 to ~135 target images, a category word, a pretrained model,
#        ~200 real images with captions close to the target caption
# Output: updated key/value projections plus one new token embedding

def personalize(target_images, category, model, reg_set):
    # new modifier token, starting from a rarely used token embedding  [app. C]
    v_star = add_token(model.tokenizer, rare_init=True)
    # trainable: key and value projections of every cross-attention layer,
    # and the new token embedding  [§3.1]
    params = cross_attention_kv(model.unet) + [embedding(v_star)]
    # step count and rate from [§3.2] and [app. E]
    opt = make_optimizer(params, lr=8e-5)
    for step in range(250):
        # half target, half real regularization samples  [app. E]
        batch = sample_balanced(target_images, reg_set, batch_size=8)
        # target samples get the new token and a random size or zoom phrase
        # with a loss mask  [app. E]
        loss = denoising_loss(model, batch)   # [eq. 2] with masks
        loss.backward()
        opt.step()
        opt.zero_grad()
    return params

Variant differences: joint training adds one token per concept and merges their batches §3.2. The closed-form merge replaces the loop with a single least-squares solve per layer and per key or value matrix eq. 4 eq. 5.

Encoder method, zero-shot path. No training happens here; the photo goes straight into the prompt conditioning §3.1 §3.2 §3.3.

# Pre-trained subject encoder, zero-shot path  [§3.1] [§3.2] [app. A.5]
# Input: one subject photo, its category word, a prompt
# Output: an image that carries the subject, with no per-subject training

def generate_zero_shot(subject_img, category, prompt, model):
    # 16 subject tokens from the multimodal query module  [§3.1]
    queries = multimodal_encoder(subject_img, category)
    # project them into the text embedding space  [§3.2]
    subject_tokens = project(queries)
    # insert position is not stated in the paper  [§3.2]
    embeds = splice(clip_text(prompt), subject_tokens)
    # PNDM sampler with classifier-free guidance 7.5  [app. A.5]
    return sample(model.unet, embeds, scale=7.5, sampler="PNDM")

The fine-tuned path replaces the direct call with the mean of the photo embeddings, then trains the U-Net for 40 to 120 steps §3.3 app. A.5.

Multi-person path, one photo per person. Only the prompt embeddings change, and the early sampling steps use the plain text §4.1 §4.3.

# Subject-augmented conditioning for several people  [§4.1] [eq. 2]
# Input: prompt tokens (n,), one photo per person (m, 3, H, W), token index
#        per person (m,)
# Output: conditioning embeddings (n, d)

def augment_prompt(prompt_tokens, photos, person_token_index):
    # Encode the prompt with the text encoder  [§3]
    text_embeds = text_encoder(prompt_tokens)
    # Encode each photo with the image encoder  [§4.1]
    photo_embeds = image_encoder(photos)
    for j, i in enumerate(person_token_index):
        # Only the person's token is replaced, by an MLP over both  [eq. 2]
        text_embeds[i] = mlp(concat(text_embeds[i], photo_embeds[j]))
    return text_embeds

Variant differences: the text-only conditioning is used while the timestep is above α·T, then this augmented conditioning §4.3 eq. 5. The released MLP is deeper than eq. 2 (see the card's Paper vs code section).

Training-free shared attention, for a set of images in one style. Only the attention call changes; no weights are trained §3.2.

# Shared self-attention for one layer; index 0 is the reference  [§3.2]
# queries, keys, values: (N, m, d); output: (N, m, d)

def shared_self_attention(q, k, v):
    # Match the targets' queries and keys to the reference statistics  [§3.2]
    q, k = adain_to_reference(q), adain_to_reference(k)
    out = []
    for i in range(N):
        # Each image attends to its own tokens and the reference's  [§3.2]
        keys = concat(k[i], k[0])
        vals = concat(v[i], v[0])
        out.append(softmax(q[i] @ keys.T / sqrt(d)) @ vals)
    return stack(out)

Contradictions

• Training steps and paper versus released code. The paper reports 250 steps for one concept and 500 for two 2212.04488 §3.2. The card's Paper vs code section records that the released configuration files use different counts, so reproduce from the paper numbers and check the card before copying any setting.

• Retrieval threshold. The paper selects regularization captions with a CLIP text similarity above 0.85 2212.04488 §3.1. The released retrieval step does not apply that threshold (see the card).

• Regularization source in the robustness study. Real retrieved images give the better KID on the main dataset set 2212.04488 tab. 3. The CustomConcept101 comparison uses generated regularization images for both this method and DreamBooth 2212.04488 app. A 2212.04488 tab. 4. Those numbers should not be compared directly with the main table.

• Table 4 bold marks. In the CustomConcept101 multi-concept rows, the printed bold marks the joint model for text-alignment, but the value printed for the closed-form merge is higher (0.763 against 0.757) 2212.04488 tab. 4.

• Baseline implementation. The DreamBooth baseline is a third-party implementation with its own learning rate and step settings 2212.04488 app. E. Head-to-head numbers reflect those choices as well as the method.

• Prompt phrase placement. The paper describes the size and zoom phrases as appended to the prompt 2212.04488 §3.2. The released code prepends them (see the card).

• Fine-tune learning rate, encoder method. The text gives 5e-5 for the fine-tune 2305.14720 §3.3; the appendix and the released config give 5e-6 2305.14720 app. A.5. The card suggests 5e-6 unless tested otherwise.

• Inference steps, encoder method. The paper uses PNDM with 100 steps 2305.14720 app. A.5. The released code defaults to 50 (see the card).

• Subject-prompt dropout, encoder method. The paper describes dropping the subject prompt in 15% of pre-training steps 2305.14720 §3.2. The released code has no such step, but it also omits the pre-training stage (see the card).

• Zero-shot against fine-tuned Textual Inversion, encoder method. The text says zero-shot beats fine-tuned Textual Inversion 2305.14720 §4.2. The table agrees on DINO and CLIP-T, but on CLIP-I it gives 0.779 against 0.780 2305.14720 tab. 1.

• DreamBooth numbers across the two papers. The tuning card's table reports a third-party DreamBooth with KID and text/image alignment 2212.04488 tab. 1. The encoder card's table reports DINO, CLIP-I and CLIP-T 2305.14720 tab. 1. The metric sets and baselines differ, so the numbers should not be compared across the two cards.

• SINE guidance window. The paper applies the fine-tuned term for t > K 2212.04489 §3.2. The sampler's default arguments apply it for small t instead, while the notebook and README use the paper's window (see the card's Paper vs code section). Copy the window from the notebook.

• DreamPose pose window. The paper conditions on five consecutive poses 2304.06025 §4.2.2. The released training code fills the window with one pose repeated five times (see the card). A model trained from that code does not see the input the paper describes.

• DreamPose frame size. The paper trains at 512 × 512 [arxiv:2304.06025 supp. §1] and evaluates at 256 px 2304.06025 §5.1.1. The released code hard-codes 640 × 512 (see the card).

• StyleAligned reference keys. The paper writes the targets' keys as [K_r; K̂_t] 2312.02133 §3.2. The released code also appends the reference's keys to its own attention, which the paper does not describe (see the card). The reference's output is unaffected by this, per the card's reading, but the code was not run.

• Delayed-conditioning switch, multi-person method. Eq. 5 switches on the timestep 2305.10431 eq. 5. The released sampler switches on the step index, and its shipped script starts subject conditioning at an absolute step, so the number of text-only steps differs from α·T (see the card's Paper vs code section). Count the steps from the code, not the equation.

• Speed-up claims, multi-person method. The introduction gives 300× to 2,500× 2305.10431 §1 and the experiments 300× to 1,200× 2305.10431 §5.2. Ratios computed from the authors' tables are 394× to 1,250× for one subject and 394× to 2,499× for several 2305.10431 tab. 1 2305.10431 tab. 2. The paper's own ranges match neither exactly.

• Identity metric, multi-person method. The paper uses FaceNet similarity of detected faces 2305.10431 §5.1. The released evaluation uses a VGGFace2-trained model instead (see the card). Its identity scores are face scores and are not directly comparable to the DINO, KID or CLIP numbers in the other cards.

• Baseline coverage, multi-person method. Section 5.3 cites DreamBooth among the optimization baselines, but the multi-subject table has no DreamBooth row 2305.10431 §5.3 2305.10431 tab. 2.

• Conditioning MLP, multi-person method. Eq. 2 is a single MLP. The released code uses residual blocks with normalization, so a literal reading of eq. 2 will not match the released checkpoint 2305.10431 §4.1 2305.10431 eq. 2.

Papers

• Multi-Concept Customization of Text-to-Image Diffusion: cross-attention key and value fine-tuning with a new token, joint training and a closed-form merge for several concepts. The authors' code is non-commercially licensed; reimplement from the paper for commercial use.
• BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing: a pre-trained multimodal subject encoder that gives zero-shot subject generation and fine-tunes in tens of steps. The released code covers fine-tuning only, under BSD-3-Clause.
• FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention: one reference photo per person, embedded with CLIP and inserted into the text tokens, with a cross-attention localization loss in training and delayed subject conditioning at sampling. Persons only; the authors' results are on their own test sets. The code is MIT-licensed.
• SINE: SINgle Image Editing with Text-to-Image Diffusion Models: fine-tunes Stable Diffusion on one picture with random patch crops and a positional code, then edits by mixing the fine-tuned and pretrained predictions in early steps. Results are qualitative. The code is MIT-licensed.
• DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion: per-subject tuning of a pose-conditioned Stable Diffusion for pose-guided fashion video, with separate image and pose guidance weights. The code is MIT-licensed.
• Style Aligned Image Generation via Shared Attention: training-free shared self-attention for a set of SDXL images in one style. The code is Apache-2.0.