--- kind: concept slug: internal-learning title: Internal learning summary: >- Training a model on the single input it will process, using augmented copies or patches of that input, not a dataset. Covered: a CLIP-guided editor, a CLIP-guided stylizer and an internal GAN for blur kernels. Trade-offs: training cost for every input, no reuse across inputs, and dependence on pretrained models in the CLIP variants. tags: [image-editing, single-image, clip, per-input-training, style-transfer, blur-kernel-estimation] papers: ["2204.02491", "2112.00374", "1909.06581"] related: [text-guided-image-editing, vision-language-pretraining, test-time-training, neural-style-transfer, image-super-resolution, generative-adversarial-network] status: draft updated_at: 2026-11-01 --- ## Overview Internal learning fits a model to one input and then uses that fitted model on the same input. No training set is involved. The learned model stands in for the knowledge a dataset would normally give it. The covered methods share that setup but differ in what they output. Text2LIVE adds an edit on top of the input [arxiv:2204.02491 §3.1]. CLIPstyler restyles the input while keeping its layout [arxiv:2112.00374 §3.1]. KernelGAN trains a downscaler on the input's own patches and reads a blur kernel from it [arxiv:1909.06581 §2] [§4.2]. In each case the input supplies the training signal, and a frozen pretrained model, or the input's own statistics, judges the result. Other internal-learning methods are not covered until their cards exist. ## Variants ### Per-input edit generator (Text2LIVE) For each input, a small U-Net is trained from nothing. The only data is one image, augmented, and a text prompt [arxiv:2204.02491 §3.1] [app. A.1]. The loss is computed with a frozen CLIP model, so no task-specific network is trained beyond the generator [arxiv:2204.02491 §3.1]. The generator outputs a colour and opacity layer that is composited over the original [arxiv:2204.02491 eq. 1]. The paper's image setting runs 1000 iterations with MADGRAD at learning rate 2.5e-3, and reports about nine minutes at 512 x 512 on one RTX 6000 [arxiv:2204.02491 app. A.3]. The video setting trains one generator per layered-atlas layer, using a frozen pretrained atlas model of that video [arxiv:2204.02491 §3.2] [app. A.4]. The paper reports about an hour per layer for 70 frames [arxiv:2204.02491 app. A.3]. Card: [Text2LIVE: Text-Driven Layered Image and Video Editing](/papers/2204.02491). ### Per-input style generator with patch loss (CLIPstyler) For each content image and each text prompt, a small U-Net is trained from scratch. Its output is scored by frozen CLIP and VGG-19 losses. The whole-image loss compares the output with the content image in CLIP space, along a direction defined by a source and a target prompt [arxiv:2112.00374 eq. 2]. A patch loss scores 64 random 128-pixel crops, each with a perspective augmentation, and zeroes any patch whose loss is at or below 0.7 [arxiv:2112.00374 §4.1] [eq. 3] [eq. 4]. A VGG-19 feature loss keeps the content [arxiv:2112.00374 §4.1]. The paper trains for 200 iterations with Adam at 5 x 10^-4, which it reports as about 40 seconds per text on one RTX 2080 Ti [arxiv:2112.00374 §4.1]. No style image is needed [arxiv:2112.00374 §1]. Card: [CLIPstyler: Image Style Transfer with a Single Text Condition](/papers/2112.00374). The paper also has a fast variant that trains one decoder on many DIV2K crops, then stylises a new photo in under 0.5 seconds [arxiv:2112.00374 §5.1] [app. A]. That variant learns across images, so it is not internal learning and is not covered further here. ### Per-image internal GAN for blur kernel (KernelGAN) For each low-resolution image, two small convolutional networks are trained on that image alone, with no training data [arxiv:1909.06581 §2] [§5.2]. The generator has no activations, so its stacked filters collapse into one kernel, which is read out from its weights [arxiv:1909.06581 §4.1] [§4.2]. The discriminator scores local patches of the downscaled output against patches of the input. The paper runs 3,000 steps, then gives the estimated kernel to a non-blind super-resolution method [arxiv:1909.06581 §2] [§5.1]. The authors report 61 seconds per image on a V100 and 102 seconds on a K80, against 45 minutes for the earlier estimator they compare with [arxiv:1909.06581 §5.3] [§1]. Card: [Blind Super-Resolution Kernel Estimation using an Internal-GAN](/papers/1909.06581). ## When to use which | Situation | Use | Why | |---|---|---| | One real image, one object to change, no masks and no editing model trained for that object type | Per-input edit generator (Text2LIVE) | The method trains only on the input and needs no mask [arxiv:2204.02491 §1] [§3.1] | | One image, restyle from a text prompt, no style image, and about 40 s of GPU time per image and prompt is acceptable | Per-input style generator (CLIPstyler) | Trains from the text alone and reports the best CLIP score and user-study style score among its comparisons [arxiv:2112.00374 §4.1] [tab. 1] [tab. 2] | | Restyling must take under a second per input | Fast variant of CLIPstyler, not this concept | The per-input method needs about 40 s; the fast decoder infers in under 0.5 s but trains on a dataset [arxiv:2112.00374 §5.1] [app. G] | | Many inputs to process, or a latency budget per input | Not these variants | Every input is trained from scratch, so cost is paid per input [arxiv:2204.02491 app. A.3] [arxiv:2112.00374 app. G] [arxiv:1909.06581 §5.3] | | Need a blur kernel for one low-resolution image before super-resolution | Per-image internal GAN (KernelGAN) | Needs no training data and gives a kernel for a non-blind SR method [arxiv:1909.06581 §1] [§5.1] | | The blur changes across the image | KernelGAN as published does not cover it | The method assumes one kernel for the whole image [arxiv:1909.06581 §5.3] | | Video | Only with a pretrained layered atlas of that video | The video path depends on a frozen atlas model [arxiv:2204.02491 §3.2] [app. A.4] | | Need to add a new object, not restyle one | Look elsewhere | The paper calls the method unsuited to creating new objects [arxiv:2204.02491 §4.5] | | Prompt wording is loose or varies | Expect different edit styles | Similar prompts can give different styles; more specific wording helps [arxiv:2204.02491 §4.5] [arxiv:2112.00374 app. E] | | Reusing the reference code in a commercial product | Check the licence first | The KernelGAN code is licensed for academic and non-commercial use [code: LICENSE.txt#L7]; the CLIPstyler card flags a repository ownership question [arxiv:2112.00374 §4.1] | ## Implementation sketch The common core of the per-input variants: fit a fresh model to augmented copies of one input, using a loss from a frozen reference model. The sketch below is Text2LIVE's loop. The CLIPstyler and KernelGAN loops have different losses; see their cards for pseudocode. ```pseudocode # Per-input training of an edit layer [arxiv:2204.02491 §3.1] [app. A.3] # Input: source image I_s, target text T # Output: generator G that maps an image to a colour layer C and opacity alpha def train_edit_generator(I_s, T): # Fresh generator for this one input [arxiv:2204.02491 §3.1] G = build_unet() opt = madgrad(G.parameters(), lr=2.5e-3) for step in range(1000): # [arxiv:2204.02491 app. A.3] # Augmented crop of the input and a template-filled text # [arxiv:2204.02491 app. A.2] I_aug = augment_image(I_s) T_aug = fill_template(T) # Composite the predicted layer over the crop [arxiv:2204.02491 eq. 1] C, alpha = G(I_aug) I_o = alpha * C + (1 - alpha) * I_aug # CLIP-space loss toward the target text [arxiv:2204.02491 eq. 3] loss = clip_loss(I_o, T_aug) # Sparsity keeps the edit local [arxiv:2204.02491 eq. 7] loss = loss + sparsity(alpha) loss.backward() opt.step() # Periodically train on the un-augmented input [arxiv:2204.02491 app. A.3] if step % 75 == 0: add_source_example(I_s, T) return G ``` The paper also adds a structure loss, a screen loss and a bootstrap term from a relevancy map, which are left out here [arxiv:2204.02491 eq. 2] [eq. 4] [eq. 5] [eq. 8]. ## Contradictions Between papers: none known. The three report different quantities, such as edit quality, a CLIP score with a user study, and kernel PSNR and SSIM, so their numbers should not be compared directly. Within papers, where the text and the equations or code disagree: - CLIPstyler's prose says patches with high losses are rejected. Its equation 4 and its code zero the patches at or below the threshold instead [arxiv:2112.00374 §1] [§3.2] [eq. 4]. Implement the equation. - KernelGAN's equation 2 is printed with a sign that reads the opposite way to its text. The official code follows the text [arxiv:1909.06581 eq. 2] [§2]. - KernelGAN's text gives its x2 gain over the best non-blind baseline as 1 dB in one place and 1.1 dB in another. The table gives 1.17 dB [arxiv:1909.06581 §5.3] [tab. 1]. - Text2LIVE's paper and its released code differ in several training values, such as the sparsity weights and the form of the structure loss. The paper's stated values should not be assumed to match the shipped code. Check the card's Paper vs code section, [Text2LIVE](/papers/2204.02491), before implementing. ## Papers - [Text2LIVE: Text-Driven Layered Image and Video Editing](/papers/2204.02491): trains a per-input edit generator with CLIP losses, for images and, through a layered atlas, for video. - [CLIPstyler: Image Style Transfer with a Single Text Condition](/papers/2112.00374): trains a per-image U-Net with CLIP directional and patch losses to restyle an image from a text prompt. - [Blind Super-Resolution Kernel Estimation using an Internal-GAN](/papers/1909.06581): trains a linear-generator GAN on one low-resolution image to estimate its blur kernel.