SINE: SINgle Image Editing with Text-to-Image Diffusion Models

In plain words

You have one picture, say a painting or a photo of your dog, and you want an edit that keeps the scene but changes something, such as the style or the background. A large text-to-image model can do this only if it first learns your picture. Teaching it on that single image makes it memorise the picture, so it stops following new words and tends to copy the original too closely.

SINE keeps the large model as it is and trains a second, memorising copy on the one picture. During the first, rough steps of generating a new image, the memorising copy helps set up the content, while the large model supplies the details that the new words ask for. Later steps use the large model alone. The copy is also trained on random crops, each with a note of where the crop sits in the picture, which helps it generate images of different shapes and sizes without repeating objects.

TL;DR

• SINE edits a single real image with a text prompt, using a pretrained Stable Diffusion model fine-tuned on that image §3.1 §4.
• Early denoising steps mix the fine-tuned model's prediction into the pretrained one; late steps use the pretrained model alone §3.2.
• Patch-based fine-tuning with a positional code lets the edit run at resolutions other than the training size §3.3.
• Evidence is qualitative images, plus two parameter-sweep plots without numbers in the text §4.1 §4.3 fig. 8.
• Relevant if you need single-image, identity-preserving edits and can afford per-image fine-tuning.

Problem

Fine-tuning a text-to-image diffusion model on one image overfits it. The copy reconstructs the source well but no longer responds to other prompts (language drift) §1 §3. Direct fine-tuning also ties the model to the training resolution: at a larger size, objects get duplicated or stretched §3.3 §4.2 fig. 5. Multi-image methods (DreamBooth, Textual Inversion) assume several photos of the same subject, which a single painting does not have §1 §2.

Method

The approach has two parts, stated in the order of the paper §3:

1. Model-based classifier-free guidance §3.2. The authors take Stable Diffusion's classifier-free guidance and add a second, fine-tuned model. With step index t and threshold K, the sampler uses the mixed prediction of eq. 3 while t > K. For t ≤ K it uses eq. 2, the pretrained model alone. The fine-tuned model only fixes the early, coarse structure.
2. Patch-based fine-tuning §3.3. Each training sample is a random crop of the source image, resized to the model's native size. The denoiser also receives a positional code for the crop's location. At sampling time the code for the whole image is given, and a noise tensor of any size can be denoised. The authors describe the code as a normalised Fourier embedding of the crop's coordinates §3.3.

The base model is a latent diffusion model: an autoencoder maps images to latents, and a denoiser predicts the noise in latent space, conditioned on text through a text encoder §3.1. The fine-tuning prompt has the form "a photo/painting of a [] [class noun]", with [] a rare identifier and the class noun a coarse word such as "castle" §3.1. The code uses a different template; see Paper vs code.

Pseudocode for the two stages follows. It is our paraphrase. Eq. 1 is the latent diffusion loss, eq. 2 the classifier-free guidance, and eq. 3 the SINE mix.

# Patch-based fine-tuning on one image  [§3.3] [§4]
# Input:  source image (H, W, 3); caption; denoiser and text encoder
# Output: fine-tuned denoiser that also takes a patch position code

def finetune_step(image, caption):
    # Sample a patch of the source  [§3.3]
    # Side lengths are uniform in [0.1 H, H] and [0.1 W, W]  [§4]
    patch, area = random_crop(image, min_frac=0.1, max_frac=1.0)
    # Resize the patch to the model's native 512 x 512  [§4]
    latent = encoder(resize(patch, 512))
    # Draw a noise level and noise the latent  [§3.1]
    noisy_latent, noise, t = add_noise(latent)
    # Denoiser sees the text and the positional code F(area)  [§3.3]
    pred = denoiser(noisy_latent, t, text_encoder(caption), F(area))
    # Noise-prediction loss, eq. 1  [eq. 1]
    return mse(pred, noise)

# Sampling with model-based guidance  [§3.2] [eq. 2] [eq. 3]
# Input:  noise z_T at the target size; target prompt; fine-tuning prompt
# Output: edited latent z_0; decode it to get the image

def edit(z_T, prompt, ft_prompt, K, v, w):
    # Start from noise at the output resolution  [§3.3]
    latent = z_T
    # Walk the sampler's timesteps from high to low noise  [§3.2]
    for t in timesteps_descending:
        # Early steps (t > K): mix both models  [eq. 3]
        if t > K:
            # Pretrained conditional prediction
            eps_pre = pretrained(latent, t, prompt)
            # Fine-tuned prediction, with the positional code of the full image
            eps_ft = finetuned(latent, t, ft_prompt, full_image_code)
            # Pretrained unconditional prediction
            eps_null = pretrained(latent, t, "")
            # Mix the two conditional terms with weight v, then guide with w
            eps = w * (v * eps_pre + (1 - v) * eps_ft) + (1 - w) * eps_null
        # Late steps (t <= K): pretrained model only  [eq. 2]
        else:
            eps = (
                w * pretrained(latent, t, prompt)
                + (1 - w) * pretrained(latent, t, "")
            )
        # One sampler update; the paper does not name the sampler  [§3.2]
        latent = sampler_step(latent, eps, t)
    return latent

Key equations, as written in the paper:

L_{LDM} = \mathbb{E}_{\mathcal{E}(\mathcal{I}), y, \epsilon, t}\left[\|\epsilon - \epsilon_\theta(z_t, t, \tau_\theta(y))\|_2^2\right] eq. 1

\tilde\epsilon_\theta(z_t, c) = w\,\epsilon_\theta(z_t, c) + (1-w)\,\epsilon_\theta(z_t) eq. 2

\tilde\epsilon_\theta(z_t, c) = w\big(v\,\epsilon_\theta(z_t, c) + (1-v)\,\hat\epsilon_\theta(z_t, \hat c)\big) + (1-w)\,\epsilon_\theta(z_t) eq. 3

In eq. 3, ε_θ is the pretrained model, ε̂_θ the fine-tuned model, c the target prompt, ĉ the fine-tuning prompt, w the classifier-free weight, and v the weight on the pretrained term. The paper gives the prediction step through Tweedie's formula §3.2. Only K and v have stated values; w does not §4 §4.3.

Hyperparameters

Results

Author-reported. Setup: one real image per experiment (free-to-use photos from Flickr and Unsplash, and famous paintings) §4.1; Stable Diffusion §4; one RTX 8000 GPU §4. Baselines: DreamBooth (an unofficial Stable-Diffusion implementation) and Textual Inversion §4.2; appendix adds ILVR, SDEdit and Prompt-to-Prompt app. A.

The paper reports no numeric headline. Appendix B and C give more visual examples and the sweep settings used for them app. B app. C.

Paper vs code

Code: the official repository at commit c0df9502b622fe36823ccf3622c8df381e801dd8 (MIT, see frontmatter). Checked: configs, the guidance sampler, the patch dataset and positional code, and the notebook and README commands.

• Guidance window is reversed in the CLI default. The paper uses the fine-tuned term for t > K §3.2. The sampler's default arguments are --range_t_max 400 --range_t_min 1 stable_txt2img_guidance.py, and the check range_t_max >= step >= range_t_min applies the fine-tuned term for 1 ≤ t ≤ 400 guidance_ddim.py. The README example uses --range_t_min 500 --range_t_max 1000 README.md, and the notebook uses K_min = 400 with range_t_max = 1000 SINE.ipynb. Those match the paper. Use the notebook or README values; the CLI default is the opposite.
• Weight v maps to the pretrained weight. Eq. 3 puts v on the pretrained term. In the code the pretrained term is weighted by cond_beta, and the fine-tuned term by cond_beta_sin = 1 - cond_beta guidance_ddim.py stable_txt2img_guidance.py. So v corresponds to cond_beta (inferred from the formulas). The CLI default is 0.5 stable_txt2img_guidance.py; the paper and notebook use 0.7 SINE.ipynb. README uses 0.4 README.md.
• Training prompt. The paper uses "a photo/painting of a [] [class noun]" §3.1. The code always uses the template "painting of a sks {}", with the class word inserted, for both photos and paintings personalized_painting.py personalized_painting.py. The literal token is "sks", not ""; the config lists "*" as the placeholder v1-finetune_patch_painting.yaml.
• Preprocessing. The paper crops the image with independent height and width fractions §4. The code first resizes the whole source to 1024×1024, which drops the aspect ratio, then draws crops on a 128×128 latent grid personalized_painting.py personalized_painting.py.
• Positional code. The paper calls it a normalised Fourier embedding of the crop's coordinates §3.3. The code builds sinusoidal codes from integer grid indices 1…128 with a centre shift of 100 positional_encoding.py, resamples the 128×128 grid to the latent shape (nearest or bilinear), and adds the result to the first hidden state of the UNet openaimodel.py. The paper does not say which resampling it used.
• Interpolation differs between configs. The painting-patch config uses nearest, the picture-patch config bilinear v1-finetune_patch_painting.yaml v1-finetune_patch_picture.yaml. The paper does not state which was used for each figure.
• Text encoder is trained too. The paper says the pretrained model is fine-tuned §3.3. The code's optimiser includes the CLIP text encoder with the UNet at the same learning rate ddpm.py v1-finetune_patch_painting.yaml v1-finetune_patch_painting.yaml.
• Sampler, step count and scale are not in the paper. The notebook uses DDIM with 100 steps and η = 0 and scale 7.5 SINE.ipynb; the README uses scale 10 README.md.
• Steps, learning rate and batch size match. 800 and 10,000 steps, base rate 1e-6, batch 1 §4 match v1-finetune_picture.yaml v1-finetune_patch_painting.yaml.
• Notebook refers to a missing config. The notebook loads v1-inference_patch_nearest.yaml SINE.ipynb, but the repository has v1-inference_patch_nearest_interp.yaml.
• README sampling uses the non-patch config. The README examples pass v1-inference.yaml as --sin_config README.md README.md, which builds a UNet with no positional code. The notebook uses the patch config for patch-trained checkpoints SINE.ipynb. Checkpoints are loaded with strict=False stable_txt2img_guidance.py, so a mismatched config loads without error.

No discrepancies found in the loss or the three equations' structure; the sampler's guidance expression matches eq. 3 once v is mapped to cond_beta.

Pitfalls

• Copy the notebook's guidance window, not the CLI default. The CLI default applies the fine-tuned term at the other end of the schedule (see Paper vs code).
• Early steps cost more than eq. 3 needs. Eq. 3 needs the pretrained conditional, the fine-tuned conditional and the pretrained unconditional. The code also runs the fine-tuned unconditional branch guidance_ddim.py; its output does not enter the update. Budget for the extra pass.
• Patch training takes about 12 times the steps of the plain fine-tune. 10,000 steps against 800 §4. Wall-clock time is not stated.
• Positional codes are added, not concatenated. An implementer who concatenates them will not reproduce the architecture.
• Sampling resamples the whole 128×128 grid to the target size. The sampling scripts never pass crop boxes stable_txt2img_guidance.py, so the full-image code is stretched to the output latent. This matches §3.3, but very different aspect ratios produce artifacts the authors themselves show (duplicated or distorted content) §4.3 app. D.
• Selection bias in the comparisons. Each comparison keeps the best of four samples per prompt §4.1, and DreamBooth is an unofficial implementation §4.2.
• Per-image tuning. K and v were chosen per edit in the appendix; the text says "optimal" settings differ by image app. C. Expect to sweep K and v per image.
• Regularisation is off. The prior-preservation loss did not help in the authors' experiments app. C; the code's regularisation weight is 0 v1-finetune_patch_painting.yaml.
• Known failure modes. Confusing prompts (e.g. a chair-shaped dog) and large changes (e.g. dog to tiger in the same pose) produce artifacts §5 app. D. Editing a face changed the sweater colour in most cases, and background lettering twisted app. D.
• Inference config may be wrong silently. strict=False loading hides a missing positional grid (see Paper vs code). An inference config comment notes that init_size: 128 may need to be 1024 v1-inference_patch_nearest_interp.yaml. The authors do not report testing that.
• Licence of the code. The repository is MIT LICENSE, but its copyright line names people who are not this paper's authors LICENSE. The MIT licence governs the code; do not assume the paper's authors hold all of its copyright.

Lineage

• Builds on: latent diffusion models, i.e. Stable Diffusion, as the base §3.1; classifier-free guidance §3.2; DreamBooth and Textual Inversion for single-subject fine-tuning §2 §3.1; positional coding for arbitrary-size GAN generation (Chai et al., cited as the inspiration for patch training) §3.3.
• Compared against: DreamBooth, Textual Inversion §4.2; ILVR, SDEdit, Prompt-to-Prompt app. A.
• Superseded by: not known from content in this wiki.