Internal learning

Overview

Internal learning fits a model to one input and then uses that fitted model on the same input. No training set is involved. The learned model stands in for the knowledge a dataset would normally give it.

The covered methods share that setup but differ in what they output. Text2LIVE adds an edit on top of the input 2204.02491 §3.1. CLIPstyler restyles the input while keeping its layout 2112.00374 §3.1. KernelGAN trains a downscaler on the input's own patches and reads a blur kernel from it 1909.06581 §2 §4.2. In each case the input supplies the training signal, and a frozen pretrained model, or the input's own statistics, judges the result.

Other internal-learning methods are not covered until their cards exist.

Variants

Per-input edit generator (Text2LIVE)

For each input, a small U-Net is trained from nothing. The only data is one image, augmented, and a text prompt 2204.02491 §3.1 app. A.1. The loss is computed with a frozen CLIP model, so no task-specific network is trained beyond the generator 2204.02491 §3.1. The generator outputs a colour and opacity layer that is composited over the original [arxiv:2204.02491 eq. 1]. The paper's image setting runs 1000 iterations with MADGRAD at learning rate 2.5e-3, and reports about nine minutes at 512 x 512 on one RTX 6000 2204.02491 app. A.3. The video setting trains one generator per layered-atlas layer, using a frozen pretrained atlas model of that video 2204.02491 §3.2 app. A.4. The paper reports about an hour per layer for 70 frames 2204.02491 app. A.3. Card: Text2LIVE: Text-Driven Layered Image and Video Editing.

Per-input style generator with patch loss (CLIPstyler)

For each content image and each text prompt, a small U-Net is trained from scratch. Its output is scored by frozen CLIP and VGG-19 losses. The whole-image loss compares the output with the content image in CLIP space, along a direction defined by a source and a target prompt 2112.00374 eq. 2. A patch loss scores 64 random 128-pixel crops, each with a perspective augmentation, and zeroes any patch whose loss is at or below 0.7 2112.00374 §4.1 eq. 3 eq. 4. A VGG-19 feature loss keeps the content 2112.00374 §4.1. The paper trains for 200 iterations with Adam at 5 x 10^-4, which it reports as about 40 seconds per text on one RTX 2080 Ti 2112.00374 §4.1. No style image is needed 2112.00374 §1. Card: CLIPstyler: Image Style Transfer with a Single Text Condition.

The paper also has a fast variant that trains one decoder on many DIV2K crops, then stylises a new photo in under 0.5 seconds 2112.00374 §5.1 app. A. That variant learns across images, so it is not internal learning and is not covered further here.

Per-image internal GAN for blur kernel (KernelGAN)

For each low-resolution image, two small convolutional networks are trained on that image alone, with no training data 1909.06581 §2 §5.2. The generator has no activations, so its stacked filters collapse into one kernel, which is read out from its weights 1909.06581 §4.1 §4.2. The discriminator scores local patches of the downscaled output against patches of the input. The paper runs 3,000 steps, then gives the estimated kernel to a non-blind super-resolution method 1909.06581 §2 §5.1. The authors report 61 seconds per image on a V100 and 102 seconds on a K80, against 45 minutes for the earlier estimator they compare with 1909.06581 §5.3 §1. Card: Blind Super-Resolution Kernel Estimation using an Internal-GAN.

When to use which

Implementation sketch

The common core of the per-input variants: fit a fresh model to augmented copies of one input, using a loss from a frozen reference model. The sketch below is Text2LIVE's loop. The CLIPstyler and KernelGAN loops have different losses; see their cards for pseudocode.

# Per-input training of an edit layer  [arxiv:2204.02491 §3.1] [app. A.3]
# Input:  source image I_s, target text T
# Output: generator G that maps an image to a colour layer C and opacity alpha

def train_edit_generator(I_s, T):
    # Fresh generator for this one input  [arxiv:2204.02491 §3.1]
    G = build_unet()
    opt = madgrad(G.parameters(), lr=2.5e-3)
    for step in range(1000):  # [arxiv:2204.02491 app. A.3]
        # Augmented crop of the input and a template-filled text
        # [arxiv:2204.02491 app. A.2]
        I_aug = augment_image(I_s)
        T_aug = fill_template(T)
        # Composite the predicted layer over the crop  [arxiv:2204.02491 eq. 1]
        C, alpha = G(I_aug)
        I_o = alpha * C + (1 - alpha) * I_aug
        # CLIP-space loss toward the target text  [arxiv:2204.02491 eq. 3]
        loss = clip_loss(I_o, T_aug)
        # Sparsity keeps the edit local  [arxiv:2204.02491 eq. 7]
        loss = loss + sparsity(alpha)
        loss.backward()
        opt.step()
        # Periodically train on the un-augmented input  [arxiv:2204.02491 app. A.3]
        if step % 75 == 0:
            add_source_example(I_s, T)
    return G

The paper also adds a structure loss, a screen loss and a bootstrap term from a relevancy map, which are left out here 2204.02491 eq. 2 eq. 4 eq. 5 eq. 8.

Contradictions

Between papers: none known. The three report different quantities, such as edit quality, a CLIP score with a user study, and kernel PSNR and SSIM, so their numbers should not be compared directly.

Within papers, where the text and the equations or code disagree:

• CLIPstyler's prose says patches with high losses are rejected. Its equation 4 and its code zero the patches at or below the threshold instead 2112.00374 §1 §3.2 eq. 4. Implement the equation.
• KernelGAN's equation 2 is printed with a sign that reads the opposite way to its text. The official code follows the text 1909.06581 eq. 2 §2.
• KernelGAN's text gives its x2 gain over the best non-blind baseline as 1 dB in one place and 1.1 dB in another. The table gives 1.17 dB [arxiv:1909.06581 §5.3] tab. 1.
• Text2LIVE's paper and its released code differ in several training values, such as the sparsity weights and the form of the structure loss. The paper's stated values should not be assumed to match the shipped code. Check the card's Paper vs code section, Text2LIVE, before implementing.

Papers

• Text2LIVE: Text-Driven Layered Image and Video Editing: trains a per-input edit generator with CLIP losses, for images and, through a layered atlas, for video.
• CLIPstyler: Image Style Transfer with a Single Text Condition: trains a per-image U-Net with CLIP directional and patch losses to restyle an image from a text prompt.
• Blind Super-Resolution Kernel Estimation using an Internal-GAN: trains a linear-generator GAN on one low-resolution image to estimate its blur kernel.