HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models

In plain words

Teaching an image generator a specific person's face usually means retraining part of the model for several minutes, and the result is a file of about a gigabyte. This paper trains a helper network that looks at one photo of a face and writes out a very small set of model changes for that face in one step. The helper is only a starting guess, so a short extra round of training on the same photo fills in the fine details.

The small set of changes is what makes the approach cheap: a personal model becomes a file of about 120 KB instead of a full copy of the model. The method keeps the generator's general knowledge of styles, so the same person can be drawn in different styles. The authors report that it reaches comparable likeness to the slow method while being far faster. The work covers faces only.

TL;DR

• One face photo goes into a hypernetwork (a ViT-H encoder with a transformer decoder), which predicts a small set of low-rank weight residuals for Stable Diffusion v1.5 §4.2 §5.
• The residuals live in a "lightweight DreamBooth" space: a rank-1 LoRA with two frozen random projections (widths a = 100 and b = 50) leaves about 30K learned values §4.1.
• A 40-step fast fine-tune, with the rank raised above 1, adds identity detail §4.3.
• Author-reported: about 20 seconds per face, 25x faster than DreamBooth, 125x faster than Textual Inversion, and a personal model about 10,000x smaller than DreamBooth [abstract].
• No official code was found. Training details (optimizer, learning rate, batch size, hypernetwork training length) are not stated.

Problem

Optimisation-based personalisation fine-tunes all or most of a text-to-image model per subject. The paper states that DreamBooth on Stable Diffusion updates a UNet of over 1 GB, and that one subject takes about 5 minutes for about 1K iterations §3 §1. The paper notes that LoRA is more memory-efficient and faster than full DreamBooth, and uses rank-1 LoRA as its starting point §3. The authors want to keep DreamBooth's style diversity and subject fidelity while cutting both the size and the time per subject §1.

Method

The method has three parts. (1) Lightweight DreamBooth (LiDB) defines a low-dimensional weight space for one subject. (2) A hypernetwork predicts LiDB weights from one face image. (3) Rank-relaxed fast fine-tuning refines the predicted weights §4 fig. 2.

4.1 Lightweight DreamBooth

Start from a rank-1 LoRA residual on each linear layer. Write the down projection as A = A_\text{aux} A_\text{train} and the up projection as B = B_\text{train} B_\text{aux}. The auxiliary matrices are random, have orthogonal rows of constant magnitude, and are frozen. Only the two training matrices are learned §4.1:

\Delta W x = A_\text{aux} A_\text{train} B_\text{train} B_\text{aux} eq. 1

Shapes: A_\text{aux} \in \mathbb{R}^{n \times a}, A_\text{train} \in \mathbb{R}^{a \times r}, B_\text{train} \in \mathbb{R}^{r \times b}, B_\text{aux} \in \mathbb{R}^{b \times m}, with r \ll \min(n, m), a < n and b < m §4.1. The paper uses r = 1 and sets a = 100, b = 50, which gives about 30K trainable values and a stored size of about 120 KB §4.1 fig. 4. The learned part of one layer is a \cdot r + r \cdot b values, which is 150 when r = 1 (derived from the shapes above, not stated in the paper).

The hypernetwork predicts only the learned matrices A_\text{train} and B_\text{train} for every cross-attention and self-attention layer of the UNet and of the CLIP text encoder §4.2 §5.

4.2 HyperNetwork

The hypernetwork \mathcal{H}_\eta maps a reference image \mathbf{x} to predicted residuals \hat{\theta} = \mathcal{H}_\eta(\mathbf{x}) §4.2.

• Encoder. A ViT-H encodes the face image. The image is encoded only once, and its features are reused in every refinement round §4.2.
• Decoder. A transformer decoder with two hidden layers receives a sequence made of the image features concatenated with the current weight features. Positional embeddings model the dependencies between layers. Trainable linear layers turn the decoder outputs into layer weights §4.2 fig. 3.
• Iterative refinement. The predicted weights start at zero, \theta_0 = 0. Each round feeds the previous prediction back into the decoder, with the image features unchanged, for s rounds §4.2 eq. 3. The value of s is not stated. The ablation "Ours (k=1)" uses a single round tab. 3.
• Supervision text. Every training sample uses the prompt "a [V] face", where [V] is a rare identifier. No learned token embedding is used. At test time, prompt variants add edits, for example "a [V] face in impressionist style" §4.2.
• Training data. 15K CelebA-HQ images are used §4.2 §5. The paper reports that this is far fewer identities than E4T (100K) or InstantBooth (1.43M) need §4.2.

Training loss: a weighted sum of a diffusion term and a weight-space term. The weight-space term compares the predicted residuals with pre-optimised weights for the same image §4.2 fig. 2.

L(\mathbf{x}) = \alpha \lVert \mathcal{D}(\mathbf{x} + \epsilon, \mathbf{c}) - \mathbf{x} \rVert_2^2 + \beta \lVert \hat{\theta} - \theta \rVert_2^2 eq. 2

Here \theta are the pre-optimised weights for \mathbf{x}, \mathbf{c} is the prompt embedding, and \alpha, \beta weight the two terms (values not stated) §4.2.

The refinement rule is

\hat{\theta}_k = \mathcal{T}(\mathbf{f}, \hat{\theta}_{k-1}) eq. 3

where \mathbf{f} are the encoder features, \mathcal{T} is the decoder and k runs up to s §4.2.

# Phase 1: train the hypernetwork  [§4.2] [fig. 2] [eq. 2] [eq. 3]
# Inputs per identity: face image x and precomputed LiDB weights theta
# Output: hypernetwork H (ViT-H encoder + transformer decoder)

def train_step(images, target_thetas):
    # Encode each image once; reuse the features in every round  [§4.2]
    features = vit_h_encoder(images)           # (B, tokens, d)
    # Start the predicted weights at zero  [§4.2]
    theta_hat = zeros_like(target_thetas)
    for k in range(s):                         # s not stated  [§4.2]
        # Decoder output is turned into weights by linear layers  [fig. 3]
        theta_hat = decoder(features, theta_hat)   # [eq. 3]
    # Weight-space term plus diffusion denoising term  [§4.2] [eq. 2]
    weight_loss = squared_error(theta_hat, target_thetas)
    denoise_loss = diffusion_loss(
        unet_with(theta_hat), images, prompt="a [V] face")
    return alpha * denoise_loss + beta * weight_loss

4.3 Rank-relaxed fast fine-tuning

The hypernetwork prediction gets the overall direction right. It usually keeps semantic attributes such as gender, hair colour and skin tone, but it misses fine detail §4.3 fig. 5.

The fine-tuning step has two parts §4.3:

1. Add the predicted residuals to the base model weights.
2. Fine-tune with a LoRA whose rank is raised above the rank of 1 used in LiDB. The new rank is not stated. The paper calls this rank relaxation and says it is the key idea behind the higher fidelity.

The loss is the plain diffusion denoising loss, L(\mathbf{x}) = \lVert \mathcal{D}_{\hat{\theta}}(\mathbf{x} + \epsilon, \mathbf{c}) - \mathbf{x} \rVert_2^2, with the same "a [V] face" prompt §4.3. The paper reports that 40 iterations are enough when the hypernetwork starts the fine-tune §4.3.

# Phase 2: rank-relaxed fast fine-tuning  [§4.3] [fig. 5]
# Input: one face image x and its predicted residuals theta_hat
# Output: personalised diffusion model

def fast_finetune(image, theta_hat):
    # Add the predicted residuals to the base model weights  [§4.3]
    model = add_residuals(base_model, theta_hat)
    # LoRA with a rank above 1; the new rank is not stated  [§4.3]
    adapter = attach_lora(model, rank=new_rank)
    for step in range(40):                     # 40 iterations  [§4.3]
        loss = diffusion_loss(adapter, image, prompt="a [V] face")
        # Optimiser and learning rate are not stated  [§4.3]
        update(adapter, loss)
    return adapter

Hyperparameters

Results

Author-reported. Setup: Stable Diffusion v1.5 base model; face identities; evaluation on 100 CelebA-HQ identities and 30 prompts with the metrics defined in the DreamBooth paper §5. The baselines are DreamBooth, Textual Inversion and, in the supplement, E4T §5 [supp.]. Compare numbers inside one table only.

Headline numbers from the abstract [abstract]:

• about 20 seconds per face;
• a speed-up of 25x over DreamBooth and 125x over Textual Inversion;
• a personal model about 10,000x smaller than a DreamBooth model.

The paper does not describe the timing protocol behind these factors.

Versus shortened DreamBooth runs with the learning rate raised, where "Agg-1" uses 400 iterations and "Agg-2" uses 40 tab. 2:

Ablation of the hypernetwork components tab. 3:

User studies (preference percentages; the third column is undecided) tab. 4 tab. 5:

Supplement, E4T on 10 SFHQ identities and 20 style prompts [supp.]:

Qualitative results: a gallery of stylised outputs fig. 6 and comparisons with DreamBooth, Textual Inversion and E4T fig. 7 [supp. fig.]. The paper says its method keeps identity better than the baselines in the one-image setting, while E4T follows the reference pose and realism more closely §5.2.

Paper vs code

No official code.

Search done: the arXiv metadata has no code link. The project page (hyperdreambooth.github.io) links only the arXiv paper, with no code or model release. A GitHub search for "hyperdreambooth" returned six repositories, none owned by an author or by Google Research. A search restricted to the google-research organisation returned nothing. The third-party reimplementations found were not used.

Pitfalls

• Training recipe is missing. The paper gives no optimiser, learning rate, batch size, number of hypernetwork training steps, or hardware. An implementation has to choose all of these. What the paper does specify is the architecture, the loss shape, and the fine-tune length.
• Size figure conflict. The text gives about 120 KB for the LiDB model §4.1. The Figure 4 caption says "roughly 100KB" fig. 4. Both describe the same 30K-value model, so use 120 KB, which matches the text, and note the difference.
• Pre-optimised targets. Equation 2 needs pre-optimised weights \theta for every training image §4.2 fig. 2. How these were produced (method, number of steps, learning rate) is not stated. Building this target set is a large hidden cost that the 20-second inference time does not show.
• Denoising term. Equation 2 writes the diffusion term as a distance between \mathcal{D}(\mathbf{x}+\epsilon, \mathbf{c}) and \mathbf{x}. The paper does not say whether the network predicts \mathbf{x} or the noise \epsilon, so use the standard denoising objective of the base model and record the choice.
• Refinement rounds. The paper says the weights are refined over s rounds and that intermediate predictions feed the next round §4.2. It does not say whether the loss is applied to every round or only to the last one, nor the value of s. The Figure 3 caption describes the output as weight deltas added to the diffusion network fig. 3, while the text describes predicted weights. Treat the output as residuals added to the base weights.
• Rank relaxation. The new rank, whether the predicted residuals are frozen during the LoRA stage, and the fine-tune learning rate are not stated §4.3. The paper's claim that this beats fixed-rank LoRA depends on these values, and no ablation over rank is given.
• Speed-up arithmetic. The 25x figure is consistent with 40 fine-tune iterations against 1,000 DreamBooth iterations (1,000 / 40 = 25 §1 §3). The table caption gives 1,200 iterations for vanilla DreamBooth tab. 2, which would give 30x. The text does not explain the choice. Timings also do not say whether the hypernetwork forward pass is included.
• Textual Inversion speed. The 125x claim against Textual Inversion has no stated iteration count or hardware, so it cannot be checked from the paper.
• Evaluation overlap. The hypernetwork is trained on 15K CelebA-HQ images §4.2 §5, and the evaluation uses 100 CelebA-HQ identities §5. The paper does not say whether these identities are disjoint.
• Metric limits. The authors say face recognition is weak here, because recognition networks trained on real faces do not match stylised outputs, and that DINO and CLIP-I favour structure and semantics over fine detail §5 [supp.]. This is why the user studies were added.
• Faces only. All experiments and training data are faces, mostly synthetic SFHQ faces for visuals and CelebA-HQ for training §5. Behaviour on other subjects is not reported.
• Limitations stated by the authors [supp. Limitations]: (1) wrong high-level attributes from the hypernetwork's initial guess, (2) too little subject detail after fast fine-tuning, and (3) underfitting that gives poor editability for some styles. Out-of-distribution lighting or pose can make these worse.
• Figure and table numbers in this card follow the order of the LaTeX source. Check them against the PDF before citing externally. The supplement is a separate file, so its table and figure are cited as [supp.].

Lineage

• Builds on: DreamBooth (subject-driven full fine-tuning; the [V] identifier comes from it), LoRA (arXiv:2106.09685), and HyperStyle, whose iterative prediction scheme the decoder follows §4.2 §2.
• Compared against: Textual Inversion (arXiv:2208.01618), DreamBooth, and E4T (arXiv:2302.12228) §2 §5. Concurrent work named in the paper: InstantBooth, Taming Encoder, FastComposer, Face0, Celeb-basis §2.
• Superseded by: not known from content already in this wiki.