Text2LIVE: Text-Driven Layered Image and Video Editing

In plain words

Text2LIVE changes how an existing object looks, such as its texture, or adds an effect such as smoke or fire to a photo or a short video. You give it the image or video and a short text description of the change. It needs no separate model trained for the task and no mask drawn around the object.

It works by learning from the one input it is given. The edit is not painted straight onto the picture. Instead, the system produces a thin transparent layer that carries the new colours and how strongly each spot should show them, and that layer is placed over the original. Because the original stays underneath, the rest of the scene is kept. Pretrained image-text models judge whether the result matches the description and whether the layout is still the same. For video, the same idea is applied to a flat "atlas" of the video, so an edit made once carries through all frames consistently.

TL;DR

• Zero-shot, text-only editing of existing objects in a single image or video §1.
• The generator is trained from scratch on augmented copies of one input, and it outputs a colour-plus-opacity layer composited over the input §3.1 eq. 1.
• Losses are defined in CLIP space: a composition loss, a green-screen loss on the layer alone, and a self-similarity structure loss, plus sparsity and a relevancy-based bootstrap §3.1 eq. 2.
• Author-reported: human judges preferred the method over mask-free image baselines and over two video ablations in 73 to 86 percent of votes tab. 1.

Problem

Editing an existing object in an arbitrary real photo is hard for three reasons. GAN-based editors work only inside narrow image domains and need the input inverted into the GAN latent space §2. Diffusion-based editors trade off satisfying the edit against keeping the original content, and inpainting variants need user masks §1 §2. CLIP-guided methods that work without a pretrained generator have so far been global, for example stylisation, or rendered from scratch §2. For video, propagation-based methods need keyframes or optical flow, and per-frame layer methods cannot propagate an edit consistently over time §2.

The method assumes a pretrained, frozen CLIP model is available, and that the input is one image or video plus one target prompt. For video it additionally assumes a pretrained neural layered atlas model of that video §3.2 app. A.4.

Method

Image framework §3.1

1. The generator G takes a source image I_s and outputs an edit layer E = (C, alpha): a colour image C and an opacity map alpha. The output is I_o = alpha * C + (1 - alpha) * I_s eq. 1 §3.1.
2. The generator is trained from scratch for each input. Its training set is an internal dataset of augmented (image, text) pairs made from the one input; augmentations are crops, scaling, flips and colour jitter, and the text is composed with a randomly chosen template §3.1 app. A.2.
3. The composition loss L_comp has two parts: a cosine-distance term between CLIP embeddings of I_o and the target text T, and a directional term that compares the CLIP change from I_s to I_o with the text change from the ROI text T_ROI to T eq. 3 §3.1.
4. The screen loss L_screen places the edit layer over a green background and asks CLIP to match a template built from the target, "{T} over a green screen" eq. 4 §3.1.
5. The structure loss L_structure compares the self-similarity matrices S(I) of the K spatial tokens from CLIP's deepest transformer layer for I_s and I_o eq. 5 eq. 6 §3.1. Tokens have dimension 768 §3.1.
6. The sparsity term L_reg = gamma * ||alpha||_1 + Psi_0(alpha) pushes alpha toward zero, with Psi_0(x) = 2 * sigmoid(5x) - 1 as a smooth L0 approximation eq. 7 §3.1.
7. The bootstrap term L_init = MSE(R(I_s), alpha) starts alpha from a relevancy map R computed for T_ROI with Chefer et al., at 224 x 224. It is annealed during training, because the relevancy maps are noisy eq. 8 §3.1 §3.1 footnote.
8. The total loss is L = L_comp + lambda_g * L_screen + lambda_s * L_structure + lambda_r * L_reg, with the bootstrap term added while it is active eq. 2 §3.1 app. A.3.

# Image framework: train one generator on one input  [§3.1] [app. A.3]
# Input:  source image I_s; target text T; ROI text T_roi; screen text T_scr
# Output: generator G that maps an image to an edit layer (C, alpha)

def train_generator(I_s, T, T_roi, T_scr):
    # Fresh generator, trained from scratch on this one input  [§3.1]
    G = build_unet()
    # MADGRAD: lr 2.5e-3, momentum 0.9, weight decay 0.01  [app. A.3]
    optimizer = madgrad(G.parameters())
    for step in range(1000):  # 1000 iterations for images  [app. A.3]
        # Internal example: augmented source crop and augmented text
        # [§3.1] [app. A.2]
        I_aug = augment_image(I_s)
        T_aug = augment_text(T)  # one of 14 templates  [app. A.2.2]
        # Predict the edit layer and composite it over the crop  [eq. 1]
        C, alpha = G(I_aug)
        I_o = alpha * C + (1 - alpha) * I_aug
        # Composition loss: cosine term plus directional term  [eq. 3]
        L_comp = clip_cosine(I_o, T_aug)
        L_comp = L_comp + clip_direction(I_aug, I_o, T_roi, T_aug)
        # Screen loss: the layer over green should match T_scr  [eq. 4]
        screen = alpha * C + (1 - alpha) * GREEN
        L_screen = clip_cosine(screen, T_scr)
        # Structure loss: self-similarity of CLIP tokens  [eq. 5] [eq. 6]
        L_struct = frobenius(self_sim(I_aug) - self_sim(I_o))
        # Sparsity: L1 plus smooth L0 on alpha  [eq. 7]
        L_reg = gamma * l1(alpha) + psi_0(alpha)
        # Bootstrap: pull alpha toward the ROI relevancy map  [eq. 8]
        L_init = mse(relevancy(T_roi, I_s), alpha)
        # Weighted sum; the bootstrap weight is annealed  [eq. 2] [§3.1]
        loss = (L_comp + lam_g * L_screen + lam_s * L_struct
                + lam_r * L_reg + lam_init * L_init)
        loss.backward()
        optimizer.step()
        # Every 75 steps, also train on the un-augmented source  [app. A.3]
        if step % 75 == 0:
            add_unaugmented_example(I_s, T)
    return G

Video framework §3.2 app. A.4

1. A pretrained, frozen neural layered atlas (NLA) model supplies a mapping from video pixels to 2D atlases and a foreground opacity. The atlas is discretised to a 2000 x 2000 image §3.2 app. A.4.
2. A separate generator is trained for each atlas, foreground and background. The generator takes an atlas crop and outputs an atlas edit layer E_A §3.2 app. A.4.
3. Training crops are sampled from a short frame segment and mapped to the atlas. The edit layer and the crop are mapped back to frames by bilinear sampling with the frozen NLA mapping eq. 9 eq. 10 eq. 11 eq. 12 app. A.4.
4. The image losses in Eq. 2 are applied to the reconstructed frames, not to the atlas itself §3.2 app. A.4.
5. At inference, the whole atlas is fed to the generator, and the edit is mapped to frames and blended with the original, following the NLA procedure app. A.4.

Hyperparameters

Values are as stated in the paper unless marked as from code. Code paths are relative to the official repository at code_commit.

Results

Author-reported. Setup: 35 web images across objects, food, landscapes and other categories; seven DAVIS videos §4.1. Human preference study on Amazon Mechanical Turk with a two-alternative forced choice protocol, 82 image-text combinations and 12,450 judgments for images §4.3; 19 video-text combinations and 2,400 judgments for videos §4.3. Baselines and their settings are in §4.2. Hardware: an RTX 6000 for images (about nine minutes for 512 x 512 and 1000 iterations) and an RTX 8000 for one video layer (about 60 minutes for 70 frames and 3000 iterations) app. A.3.

Blended-Diffusion and GLIDE need user masks and are compared only qualitatively, with masks supplied §4.2 fig. 7. Ablations of the loss terms, bootstrapping and internal learning are qualitative §4.4 fig. 8. Video results are shown as representative frames fig. 6 and stated to be in the supplementary material §4.1.

Paper vs code

Searched: the arXiv text, then the project page the paper links (text2live.github.io), which links omerbt/Text2LIVE. The owner name matches the first author, Omer Bar-Tal. The repository is MIT-licensed, describes itself as the official PyTorch implementation, and was cloned at code_commit. Paper app. A.3 said code "will be made available"; this is that release. The release is not necessarily the state of the arXiv version (v2, 2022-05-25).

• Structure loss form. Paper uses the Frobenius norm of the matrix difference eq. 6 §3.1; code uses the mean squared error of the two self-similarity matrices losses.py. Because MSE averages over K² entries, lambda_s = 2 does not carry over as stated.
• Self-similarity layer. Paper says the deepest CLIP layer §3.1; code runs all transformer blocks except the last model.py model.py.
• Composition distance. Paper uses cosine distance eq. 3 §3.1; the shipped default is spherical (geodesic) distance image_config.yaml util.py. A cosine option, scaled by 1.2, exists util.py.
• Image sparsity weights. Paper: lambda_r = 5e-2 with gamma = 2, which gives L1 weight 0.1 and L0 weight 0.05 app. A.3. Shipped image config: lambda_sparsity = 6 with 0.01 on each term, so both weights are 0.06 image_config.yaml. Video matches the paper: 1e-3 L1 and 5e-4 L0 video_config.yaml.
• Screen template. Paper uses a single template §3.1; code uses two, "over a green screen." and "in front of a green screen." util.py. Green is RGB (0, 177, 64)/255, a value the paper does not state image_model.py.
• Text and CLIP-view sampling. Paper draws one template per step app. A.2.2. Code averages each CLIP loss over a random subset of the 14 template embeddings, drawn with replacement and of random size losses.py. CLIP views use horizontal flips, affine and perspective warps, colour jitter and random greyscale clip_extractor.py.
• Bootstrap schedule. Paper anneals the bootstrap weight linearly for images app. A.3. The shipped image default turns bootstrapping off (bootstrap_epoch = -1); the golden_horse example sets 1000 image_config.yaml golden_horse.yaml. The ice_cake example does not set it, so bootstrapping stays off there ice_cake.yaml. The video default is an unannealed scheduler, and car-turn_rusty sets 3000 epochs video_config.yaml car-turn_rusty.yaml.
• Video crop for background. Paper states 0.95 for video app. A.2.1. The config comment says 0.8 for background video_config.yaml. The paper does not state this.
• Video CLIP views. Image config uses 16 views image_config.yaml; video config uses 6 video_config.yaml. The paper does not state either number.
• Seed. Default seed is -1, which draws a random seed each run train_image.py image_config.yaml. Repeated runs will differ.

No other discrepancies were found in the checked hyperparameters. Learning rate, optimizer, weight decay, momentum, decay floor, iteration counts, crop, scale, flip, 14 templates, ViT-B/32 and the 75-step source insert all match.

Pitfalls

• Unstated CLIP settings. The number of augmented CLIP views and their geometric and colour augmentations are not in the paper's main text §3.1. Code values are above and in clip_extractor.py. Results depend on them.
• Structure loss scale. A Frobenius norm and a mean squared error differ in power (norm versus squared norm) and in a factor of K². Implement the form you choose and retune lambda_s, or follow the code.
• Hidden green value. The green-screen background colour is not in the paper, but the composite depends on it image_model.py.
• Bootstrapping is easy to miss. Off by default for images. Only the example configs turn it on, and at different lengths image_config.yaml. Without it, the paper reports colour bleeding around the edit §4.4 fig. 8.
• Random seed. The default seed is random train_image.py. Single runs can differ in visible quality, and the paper reports no variance across seeds.
• Relevancy depth. The code uses 10 relevancy layers image_config.yaml; the paper does not state this.
• Video needs an NLA checkpoint. The example configs point to a pretrained atlas checkpoint path car-turn_rusty.yaml. Video quality is capped by the atlas: artefacts propagate into the edit §4.5.
• Two generators per video. Foreground and background are edited separately, and the paper does not say how they are combined beyond the NLA compositing app. A.4.
• CLIP biases. Prompts can pull in new objects or shapes: "birthday cake" brings candles, and "moon" brings a crescent. More specific wording helps §4.5 fig. 9. The paper calls the method unsuited to creating new objects.
• Prompt sensitivity. Similar prompts can give different edit styles §4.5.
• Cost. About nine minutes per image on one GPU, about an hour per video layer app. A.3. Each input is trained from scratch, so there is no amortisation across inputs.
• Small evaluation. The user study covers 82 image-text pairs and 19 video-text pairs §4.3. Ablations are visual only §4.4.

Lineage

• Builds on: Neural Layered Atlases (Kasten et al.), used as a frozen video renderer §3.2; CLIP §1; Chefer et al. for relevancy maps §3.1; structure from CLIP self-similarity, following earlier work on structure-preserving losses §3.1; internal learning, as in SinGAN-style single-image methods §2.
• Compared against: Blended-Diffusion and GLIDE (mask-based), CLIPStyler, Diffusion+CLIP, VQ-GAN+CLIP, and StyleCLIP and StyleGAN-NADA in the supplement §4.2. Omnimattes are discussed as related video layering §2.
• Superseded by: not known from content in this wiki.