CLAP: Isolating Content from Style through Contrastive Learning with Augmented Prompts

In plain words

CLIP learns one space where pictures and sentences meet, and its features are very useful. The trouble is that a feature mixes two things: what the object is, and how it is presented (a photo or a sketch, a painting style, the colours). When the presentation changes, the features move, and the model can get the label wrong.

CLAP adds a small learned correction on top of CLIP's frozen features. It is trained only on sentences. Each training sentence describes a class, such as "a photo of a red car", and a changed copy is made by dropping or reordering style words, such as "a car". The correction is pushed to map the sentence and its changed copy to nearby points, and to keep the copy close to the bare class name. In effect the correction learns to ignore the style words.

Because CLIP puts sentences and pictures in one shared space, the same small correction can then be applied to picture features as well. The authors report better zero-shot and few-shot classification, and better behaviour when the wording of the prompt is changed, compared with CLIP alone.

TL;DR

• CLAP is a two-layer residual MLP placed after frozen CLIP encoders §4.2 §4.1.2.
• It is trained with an InfoNCE contrastive loss on text only: prompt versus augmented prompt, and augmented prompt versus class name eq. 4.
• Augmentations drop size, colour or order words from template prompts §4.2.1 tab. 2.
• The learned network is reused on CLIP's image encoder and text encoder at inference, with a scalar weight α on its residual branch §4.1.3 app. A.1.
• Author-reported: ViT-B/16 zero-shot average over PACS, VLCS, OfficeHome and DomainNet is 79.9% against 77.4% for CLIP tab. 3.

Problem

CLIP-like models blend content (what the object is) and style (how it looks). Under distribution shift the style part can be a spurious cue, which hurts generalisation §1. The reported symptoms include sensitivity to the wording of zero-shot prompts, weaker few-shot results, and vulnerability to adversarial inputs §1 §2. Prompt-learning methods tune prompts for each task but do not change the representation itself §2.

Image-side augmentation can isolate content in principle, but the authors argue that there are not enough realistic augmentations to change every style factor in an image §1 §4.1. Text is easier to edit for style, which motivates the text-side method §4.2.

Method

Causal view §3. The paper models vision-language data as generated from shared latent content c and style s. Content determines the label, style depends on content, and images and texts are rendered from both eq. 1. Under the assumption that augmentation changes the style variables, a contrastive objective on augmented pairs can recover the content part §3 §4.1. The authors build on earlier theory by von et al. (2021) for this step; the card does not repeat its proof.

Image augmentation variant, "Im.Aug" §4.1. This is the paper's first method and its comparison baseline.

1. Make training images with a text-to-image model from template prompts §4.1.1 tab. 1 app. B.1.
2. Apply a random crop and colour distortion to one view of each image §4.1.1 app. A.1.
3. Train the disentangled network on top of CLIP's frozen image encoder with the image InfoNCE loss eq. 2 eq. 3.
4. At inference, apply the network after the image encoder, and also after the text encoder for zero-shot classification §4.1.3.

CLAP, the text-augmentation method §4.2. This is the main method. It uses no images for training.

1. Build the training prompts from the same template as Im.Aug: each class gets 480 prompts §4.1.1 tab. 1.
2. For each prompt t, make an augmented view t̃ with the prompt operators in §4.2.1 tab. 2.
3. Encode t, t̃ and the bare class name c with the frozen CLIP text encoder. For t̃, four random (not learned) context vectors are inserted, which the paper calls IGN §4.2.1.
4. Pass the three features through the disentangled network f_c.
5. Minimise the sum of two InfoNCE terms eq. 4. The first matches t with t̃. The second matches the class name with t̃, at temperature 1.
6. After training, use f_c after CLIP's image encoder and after its text encoder §4.2 fig. 2.

# CLAP training step  [§4.2] [eq. 4]
# Writes only the disentangled network f_c; CLIP stays frozen.
# Paper form of InfoNCE  [eq. 2]; see Paper vs code for the code's form.
# Input:  prompts (B,), augmented prompts (B,), class names (B,)
# Output: scalar loss

def clap_training_loss(prompts, augmented, class_names, f_c):
    # Frozen CLIP text features  [§4.2]
    z_prompt = clip_text(prompts)          # (B, 512)
    z_aug = clip_text_with_noise(augmented)  # (B, 512)  [§4.2.1]
    z_class = clip_text(class_names)       # (B, 512)

    # Pass through the disentangled network  [§4.1.2]
    h_prompt = f_c(z_prompt)               # (B, 512)
    h_aug = f_c(z_aug)                     # (B, 512)
    h_class = f_c(z_class)                 # (B, 512)

    # Term 1: prompt against its augmented view, temperature tau  [eq. 4]
    # tau = 0.5 for PACS and VLCS, 0.3 for OfficeHome and DomainNet
    term_view = info_nce(h_prompt, h_aug, tau)

    # Term 2: class name against augmented view, temperature 1  [eq. 4]
    # lambda = 1 for PACS and VLCS, 0.0001 for OfficeHome and DomainNet
    term_class = info_nce(h_aug, h_class, 1.0)

    return term_view + lam * term_class

# Disentangled network f_c, a residual two-layer MLP  [§4.1.2]
# Input: x (B, 512) CLIP feature; output (B, 512); alpha = 1 in training
# Input and output sizes are equal for ViT-B/16, so the shortcut is identity.

def disentangled_network(x, alpha):
    # SiLU, then a normally initialised linear layer with bias  [§4.1.2]
    hidden = linear_with_bias(silu(x))        # (B, d_mid)
    # Zero-initialised linear layer, no bias; starts as identity  [§4.1.2]
    branch = zero_init_linear(hidden)         # (B, 512)
    # alpha weights the residual branch at inference only  [§4.1.2] [app. A.1]
    return alpha * branch + x

# Prompt augmentation for the text view  [§4.2.1] [tab. 2] [app. A.2]
# Each operator fires independently; probabilities for VLCS  [app. A.1]
# Input: a template prompt such as "a realistic painting of a large red car"

def augment_prompt(prompt):
    # Object Size Deletion, p = 0.5: drop one size word  [tab. 2]
    if random() < 0.5:
        prompt = drop_first_size_word(prompt)
    # Object Colour Deletion, p = 0.5: drop one colour word  [tab. 2]
    if random() < 0.5:
        prompt = drop_first_colour_word(prompt)
    # Swap Prompt Order, p = 0.5: move the content phrase first  [tab. 2]
    if random() < 0.5:
        prompt = reorder_as_object_in_style(prompt)
    # Image Type and Art Style Deletion exist, but p = 0 for VLCS
    # [app. A.1] [code: configs/train_CLAP_VLCS_ViTB.yaml#L32-L33]
    return prompt

Inference §4.1.3 §4.2 app. A.1: the image feature becomes f_c(f_x(x)) and the text feature f_c(f_t(t)). Zero-shot prediction is the cosine-similarity argmax against the class prompts. Four prompt styles are used: a bare class name, "a [class] in a photo", "a photo of a [class]", and a noised prompt with four random context vectors §5.2.1.

Prompt construction §4.1.1 tab. 1 app. B.1. Each prompt has the form "a [art style] [image type] of a [object size] [object colour] [class]". The attributes are 10 colours, 3 sizes, 8 image types and 2 art styles, which gives 480 prompts per class. Image generation for Im.Aug uses Stable Diffusion v2.1 at 512 × 512, with 480 images per class app. B.1.

Hyperparameters

Values for ViT-B/16 unless marked. Where the shipped code differs, it is noted in the next section.

Results

Author-reported. Setup: four multi-domain datasets, PACS (4 domains, 7 classes), VLCS (4 domains, 5 classes), OfficeHome (4 domains, 65 classes) and DomainNet (6 domains, 345 classes). Results are averaged over domains. Model: CLIP ViT-B/16. Baselines: CLIP and Im.Aug §5.1 tab. 3.

Few-shot curves (1, 4, 8, 16 shots) are in fig. 4; per-domain numbers are in app. A.4.2. The paper's ablations over τ, latent width and α are in fig. 6 and app. A.4.4. Ablations on prompt-operator subsets and on prompt sources are in app. A.2 and app. A.3. The paper finds the simplest prompt forms (the template and a random-style set) work best app. A.3.

Paper vs code

Code: official repository https://github.com/YichaoCai1/CLAP, linked from the paper's abstract footnote. Commit d91a68032dd8f8ae284990d1e88c2d22a0dc352a, cloned at depth 1. The repository has no LICENSE file, so it is cited by path and line only and no code is quoted.

• InfoNCE form. Paper's loss sums over cross-view pairs only and is one-directional eq. 2. Code's InfoNCELoss concatenates cross-view and same-view similarities into one 2n × 2n cross-entropy, with same-view diagonals masked. It is two-directional and uses same-view negatives loss.py. Used by the text-view term and the class-name term trainer.py.
• Im.Aug augmentation probabilities. Paper says crop and colour distortion each apply with probability 0.5 app. A.1. Code's transform applies colour jitter and the random resized crop on every call, with no probability data_utils.py. No Im.Aug training config ships.
• Reordering operator text. Paper's table shows the reordered prompt as "a large red car in a realistic painting" tab. 2. Code appends the word " style" data_utils.py.
• Drop operators. Paper describes removing style words §4.2.1. Code removes only the first matching word from each list and then stops that list data_utils.py.
• Context noise. Paper inserts zero-mean noise "into the tokenized prompts" §4.2.1. Code creates four new embedding vectors with std 0.02 and places them in the embedding sequence in place of placeholder tokens misc.py. The position (front, middle or end) is drawn once per batch, not per sample trainer.py.
• Noise scope. Paper applies noise to PACS and VLCS only app. A.1. The shipped training config is VLCS only, with noise length 4; its comment says to set it to zero when there are many classes train_CLAP_VLCS_ViTB.yaml. Configs for PACS, OfficeHome and DomainNet are not shipped.
• Few-shot optimiser. Paper gives the probe's loss, batch size, epochs and early stopping, but no optimiser app. A.1. Code uses Adam with learning rate 0.005 eval_fewshots.py.
• Adversarial step sizes. Paper gives only ε = 0.031 §5.2.3. Code sets the FGSM step to ε and the multi-step step to ε/10, with a uniform random start for multi-step attacks evaluate.py evaluate.py.
• Adversarial clamp space. Code clamps adversarial images to [0, 1] evaluate.py. Images come from CLIP's own preprocessing eval_zeroshot.py eval_zeroshot.py. CLIP's preprocessing normalises images, so the clamp and ε act on normalised tensors. This is inferred from the CLIP library's preprocessing, which is not in the repository.
• Inference α. Paper gives α per dataset app. A.1. The VLCS and OfficeHome eval configs use the same values (0.056 and 0.14), and α is applied as a multiplier on the residual branch eval_CLAP_VLCS_ViTB.yaml eval_CLAP_OfficeHome_ViTB.yaml networks.py. Training uses scale 1 train_CLAP_VLCS_ViTB.yaml.
• Matches. Adam, learning rate 1e-4, 8,000 steps, check every 480 steps, early stopping with patience 5 and delta 0.01, batch 8 and τ = 0.5 for VLCS, λ = 1 for VLCS, seed 2024, SiLU, zero-initialised projection without bias, and the 10 × 3 × 8 × 2 = 480 template count. Checked against the cited lines.
• Not checked. Hyperparameters for OfficeHome, DomainNet and the ViT-L/14 and RN50x16 runs are not in the repository's training configs. Only the VLCS training config and the VLCS and OfficeHome eval configs ship.
• Results files. The repository ships backup result text files for ViT-B and ViT-L eval_results_zeroshot_natural.back.txt. The numbers on this card come from the paper, not from those files; they were not compared against the files.

Pitfalls

• Batch duplicates in the class-name term. Term 2 uses the class name of each sample as the positive. With VLCS's 5 classes and batch size 8, the batch always holds repeated class names. The code only masks the diagonal loss.py, so identical class embeddings count as negatives for each other. The paper does not discuss this. Test it, or mask same-class pairs.
• Temperature and λ are per dataset. Values for OfficeHome and DomainNet (τ 0.3, λ 0.0001) are only in the appendix, and the shipped code has no config for them. Carry the dataset-specific values explicitly.
• α was chosen on evaluation data. The repository's α search evaluates zero-shot accuracy on the test environments search_alpha.py and scans fixed log-spaced values search_alpha.py. The paper does not say how α was chosen, so the reported gains may be optimistic. A held-out split is needed for a fair comparison.
• Noise is re-drawn each batch. The noised zero-shot prompt draws fresh Gaussian context vectors on every batch misc.py evaluate.py. Results can shift slightly between runs, even with a fixed seed.
• CUDA is hard-coded in attacks. The PGD random start and the CW one-hot target use .cuda() evaluate.py evaluate.py. CPU-only runs fail there.
• Early stopping thresholds are hard-coded. The linear probe's patience of 10 and minimum decrease of 0.001 are fixed in the code, not in the config evaluate.py evaluate.py.
• Sensitivity metrics use three prompts. The range R and std δ are over ZS(C), ZS(CP) and ZS(PC) only, not over the noised prompt §5.2.1. Do not compare them with metrics over a different prompt set.
• Compute cost. CLAP trains on text only, but needs the template prompts and a frozen CLIP. Im.Aug needs generated images, which took about 3.3 h on DomainNet against 47 min for CLAP §5.1. Those timings are author-reported on one RTX 3090.
• Section and equation numbers. These follow the LaTeX source's numbering (llncs). Appendix sections use letters. Check against the compiled PDF before you cite a number to a reader.

Lineage

• Builds on: CLIP, the frozen backbone and contrastive objective §1 §2. The causal disentanglement argument follows von et al. (2021) §3. The text-augmentation operators adapt Easy Data Augmentation (EDA) §4.2.1. The context-noise initialisation follows the prompt-learning protocol the paper cites §4.2.1; the code credits CoOp misc.py. A ControlNet-style zero-initialised layer inspires the projection §4.1.2.
• Compared against: CLIP (zero-shot and linear probe) and Im.Aug, the paper's image-augmentation variant §5.2. LLM-written and PromptStyler prompts are used as alternative training sources app. A.3.
• Superseded by: not stated in the content of this wiki.