Self-regulating Prompts: Foundational Model Adaptation without Forgetting

In plain words

Large vision-language models such as CLIP can recognise many things without any training. To make one work well on a new task, a common shortcut is to learn a few extra "prompt" vectors while the model itself stays frozen. The catch is that these prompts tend to fit the new task so closely that they drift away from what the model already knew. They then do worse on categories they were never trained on.

PromptSRC keeps the prompts close to the frozen model. It adds penalties that pull the prompted image and text features, and the predictions built from them, toward what the untouched model would produce. It also averages the prompts from several training epochs instead of keeping only the last one, and it gives the text side several phrasings per class rather than a single label. The result is meant to be a model that learns the new task while still handling unseen categories well.

TL;DR

PromptSRC is a regularised way to train CLIP prompts. Besides the normal classification loss, it adds agreement losses against the frozen model, a weighted average of the prompts across epochs, and multi-template text targets. The authors report the best base-to-novel harmonic mean among the methods they compare, with 46K tunable parameters against 3.55M for MaPLe §4.3. The code is MIT-licensed and differs from the paper in several details listed below, so check those before reproducing numbers.

Problem

Prompt learning adapts CLIP by training a small number of prompt tokens with the frozen encoders §1. Trained with a task-specific cross-entropy loss, the prompts overfit the downstream data and lose the frozen model's generalisation to new classes and datasets §1 §3. Naive remedies are either shorter training, which costs accuracy on the task itself, or changes to the architecture §3.

Method

The method builds on IVLP, an independent vision-language prompting baseline that adds learnable tokens to both encoders §3.1. Language prompts are written as learnable context before the class name; visual prompts are prepended to the image patch tokens §3.1. Prompts are deep: separate learnable sets are inserted at the transformer blocks rather than only at the input §3.1. (The number of blocks used is given in §4.1; see Paper vs code.)

Training minimises three terms, all computed for the same mini-batch §3.2:

1. Task loss. Cross-entropy on the prompted image-text similarities eq. 1.
2. Feature agreement. L1 distance between prompted and frozen image features, and between prompted and frozen text features, each with its own weight §3.2.1 eq. 2.
3. Logit agreement. A KL divergence that pulls the prompted class distribution toward the frozen CLIP class distribution §3.2.1 eq. 3.

The combined objective adds the weighted agreement terms to the task loss eq. 4 eq. 5.

The frozen targets are computed without prompts. The frozen image feature comes from the original CLIP image encoder §3.2.1. The frozen text feature for each class is the average of the text features over a pool of N prompt templates, which the authors call textual diversity §3.2.3. This average is used only as a training target; at inference the prompted text features are used, not the ensemble §3.2.3.

Prompt self-ensembling works at the epoch level. After each epoch the current prompts are added to an accumulator with a weight from a Gaussian over epoch indices, normalised so the weights sum to one §3.2.2 eq. 6 eq. 7. The accumulated prompts are used for final inference fig. 3 alg. 1.

# Self-regulating prompt training  [alg. 1]
# Input:  labelled dataset; frozen CLIP encoders f (image) and g (text)
# Output: ensembled prompts P_gpa, used for inference  [fig. 3]

def train_promptsrc(dataset, epochs, lam_img, lam_txt):
    # Frozen text targets: average over N = 60 templates  [§3.2.3]
    g_frozen = mean(g(template(c)) for template in TEMPLATES for c in classes)
    # Running sum of epoch prompts, Gaussian weighted  [§3.2.2] [eq. 7]
    P_gpa = zeros_like(P)
    for epoch in range(epochs):
        for images, labels in dataset:
            # Prompted features from the image and text encoders  [§3.1]
            f_p = f(images, P)
            g_p = g(prompt_texts(P, classes))
            # Task loss on prompted similarities  [eq. 1]
            logits = scale * normalise(f_p) @ normalise(g_p).T
            loss_ce = cross_entropy(logits, labels)
            # Frozen image feature, no prompts  [§3.2.1]
            f_frozen = f_clip(images)
            # L1 agreement on features, weighted per modality  [eq. 2]
            loss_img = lam_img * l1(f_p, f_frozen)
            loss_txt = lam_txt * l1(g_p, g_frozen)
            # KL agreement on logits, toward the frozen distribution  [eq. 3]
            loss_kl = kl_frozen_to_prompted(logits_frozen, logits)
            # Total objective  [eq. 4] [eq. 5]
            loss = loss_ce + loss_img + loss_txt + loss_kl
            P = sgd_step(P, loss)
        # Add this epoch's prompts with its normalised weight  [eq. 6] [eq. 7]
        P_gpa = P_gpa + w[epoch] * P
    return P_gpa

The pseudocode follows the paper's notation. The official code computes the KL term in a different direction and normalises the agreement losses differently; see Paper vs code.

Key equations (paper numbering follows the order in which the equations appear in the source):

• Task loss: \mathcal{L}_{CE}, minimised over the prompts P eq. 1.
• Feature agreement: \mathcal{L}_{SCL\text{-}image} = \sum_{i=1}^{d} |\tilde f_p - \tilde f| and \mathcal{L}_{SCL\text{-}text} = \sum_{i=1}^{d} |\tilde g_p - \tilde g| eq. 2.
• Logit agreement: \mathcal{L}_{SCL\text{-}logits} = D_{KL}(\mathrm{sim}(\tilde f_p, \tilde g_p) \,\|\, \mathrm{sim}(\tilde f, \tilde g)) eq. 3.
• Combined agreement: \mathcal{L}_{SCL} = \lambda_1 \mathcal{L}_{SCL\text{-}image} + \lambda_2 \mathcal{L}_{SCL\text{-}text} + \mathcal{L}_{SCL\text{-}logits} eq. 4.
• Total: \mathcal{L}_{final} = \mathcal{L}_{CE} + \mathcal{L}_{SCL} eq. 5.
• Prompt aggregation: P^{AP} = \sum_t \frac{w_t P}{\sum_i w_i} eq. 6.
• Moving-average form used in practice: P^{GPA} = \sum_i w_i P_i eq. 7.

Aggregation weights are described as samples from w_i \sim \mathcal{N}(\mu, \sigma^2) with \sum_i w_i = 1 §3.2.2. Mu is an epoch index, so the weights peak at about epoch \mu §3.2.2 app. A.

Hyperparameters

Results

Author-reported. Setup: CLIP ViT-B/16, 11 image recognition datasets following CoOp and CoCoOp, results averaged over 3 runs §4.1 app. A. Base-to-novel: the model is trained on half of the classes of each dataset and tested on both halves §4.1 §4.3.

Base-to-novel, average over 11 datasets tab. 1

Base-to-novel, PromptSRC per dataset tab. 1

The text reports gains of +1.42 HM over MaPLe and +8.26 over CLIP on the 11-dataset average §1. It also reports that PromptSRC beats MaPLe on 8 of 11 datasets with 46K tunable parameters against MaPLe's 3.55M §4.3.

Few-shot §4.4 fig. 4. Averaged over 11 datasets, the gain over MaPLe is +3.05, +2.72, +2.59, +1.80 and +1.07 points for K = 1, 2, 4, 8 and 16 shots. The per-shot values are in fig. 4 only; the per-dataset table is in App. B.

Cross-dataset tab. 3. Trained on ImageNet, evaluated on ten other datasets. PromptSRC: 71.27 on ImageNet and 65.81 averaged over the ten targets. Averages over the targets: CoOp 63.88, CoCoOp 65.74, MaPLe 66.30.

Domain generalisation tab. 4. Trained on ImageNet, tested on ImageNet-V2, ImageNet-Sketch, ImageNet-A and ImageNet-R.

Ablations (11-dataset averages, base / novel / HM).

Feature-distance choice for the agreement loss, on top of IVLP tab. 5: cosine 79.17 HM, MSE 79.33, L1 79.43 (L1 is used). Ensembling choice tab. 6: exponential moving average 79.47 HM, equal averaging 79.83, GPA 79.97. The GPA row of tab. 6 is the full model, not the isolated GPA step; the isolated value is 79.70 in tab. 2.

Compute tab. 7, SUN397, 10 epochs on one A100. Training GFLOPs: CoCoOp 162.5, IVLP 162.8, PromptSRC 179.6. Training time: IVLP 12.01 min, PromptSRC 13.13 min, CoCoOp 39.53 min. Inference GFLOPs: PromptSRC 162.8, the same as IVLP. Throughput: IVLP and PromptSRC 1380 FPS, CoCoOp 15.08 FPS tab. 7. The text says training time is 9.3% longer than IVLP §4.7.

GPA hyperparameters tab. 8 are listed in Hyperparameters above.

Paper vs code

Code: https://github.com/muzairkhattak/PromptSRC, linked from the abstract. Author-owned. Commit bb95c77b634d63488f2cad81ff4a72d53bdd06d5. MIT license (2023, Muhammad Uzair Khattak) LICENSE.

• GPA weights are not sampled. The paper describes weights drawn from a Gaussian §3.2.2. The code evaluates the Gaussian density at epochs 1..E, then divides by the sum, so the weights are deterministic promptsrc.py promptsrc.py.
• GPA timing. Algorithm 1 applies the ensemble every c iterations and adds w_i P without normalising alg. 1. The code applies it once per epoch, at the last batch promptsrc.py, with normalised epoch weights.
• GPA scope. Eq. 7 averages the prompts P eq. 7. The code averages the whole model state dict promptsrc.py. The CLIP weights are frozen, so their weighted sum is unchanged, but the accumulator holds a full model copy.
• Name of the GPA second parameter. The paper and tab. 8 call it \sigma^2. The code passes the value as the standard deviation \sigma promptsrc.py. Use the code's meaning when reproducing.
• Scale of the agreement losses. Eq. 2 sums the absolute differences over the feature dimension eq. 2. The code takes the mean over all elements promptsrc.py. This changes the effective \lambda_1 and \lambda_2 by a factor of the feature dimension.
• KL direction and normalisation. Eq. 3 writes the divergence with the prompted distribution first eq. 3. The code calls F.kl_div(log_softmax(prompted), log_softmax(frozen), log_target=True), which weights by the frozen distribution, so it computes KL(frozen ‖ prompted), and divides the sum by the number of logit entries promptsrc.py.
• Frozen text targets. The code averages the unnormalised text features of each template, then normalises the mean promptsrc.py promptsrc.py. The paper averages the template features §3.2.3.
• Templates. The paper says 60 templates are "randomly selected" from the CLIP list app. A. The code uses a fixed 60-entry list copied from the CLIP ImageNet notebook imagenet_templates.py with no random selection.
• Optimiser schedule. The paper fixes the learning rate at 0.0025 §4.1. The configs use SGD with cosine decay and one constant warm-up epoch at 1e-5 vit_b16_c2_ep20_batch4_4+4ctx.yaml.
• Cross-dataset script. scripts/promptsrc/xd_train.sh and xd_test.sh set CFG to vit_b16_c2_ep5_batch4_4+4ctx_cross_datasets xd_train.sh. The repository has no such file; the closest config is ..._ep20_..._cross_datasets.yaml, which runs 20 epochs as the paper states §4.1. The script will not find its config as written.
• Prompt depth. §3.1 says the prompts are inserted at every transformer block §3.1. §4.1 says 9 layers (base-to-novel, few-shot) or 3 layers (cross-dataset, domain generalisation) §4.1. The configs follow §4.1 vit_b16_c2_ep20_batch4_4+4ctx.yaml.
• Text-prompt initialisation. The code initialises the text context from the word embeddings of "a photo of a" when there are at most four context tokens promptsrc.py. The paper says this for the first layer only §4.1; the code's handling of deeper layers is not checked here.

Pitfalls

• Teacher copies cost memory. The trainer loads a frozen CLIP copy on the GPU in fp32 to compute the frozen text targets promptsrc.py promptsrc.py, and a second copy for the frozen image encoder. The GPA accumulator holds another full model state dict promptsrc.py. The paper's compute analysis counts only the single extra image forward pass §4.7.
• Compute claim. The text says PromptSRC's training GFLOPs are "only 0.13x higher" than IVLP §4.7. Table 7 gives 179.6 vs 162.8, which is about 10% more tab. 7. Use the table.
• Internal inconsistency in the ablation text. §4.2 says \mathcal{L}_{SCL} raises novel accuracy by 3.95 points §4.2. Table 2 gives 75.38 − 71.79 = 3.59 tab. 2. Treat the table as the source of truth.
• Base-to-novel shots are not stated in the paper. The code uses 16 shots base2new_train.sh.
• Precision. The paper does not state a precision. The config uses fp16 vit_b16_c2_ep20_batch4_4+4ctx.yaml.
• Batch size and augmentation are not in the paper. The config uses batch size 4 and random resized crop plus horizontal flip vit_b16_c2_ep20_batch4_4+4ctx.yaml.
• Gaussian weights depend on the number of epochs. Using the code's formula, base-to-novel (20 epochs, μ = 15, std 1) puts nearly all weight on epochs near 15, while few-shot (50 epochs, μ = 30, std 30) spreads it widely. Keep μ and the spread in epoch units, and normalise over the epochs that actually run.
• Reporting. Results are averages over three runs §4.1. The tables give no standard deviations tab. 1 tab. 2.

Lineage

• Builds on the IVLP baseline described by ViFi-CLIP, which the paper cites as its base §3.1 §2. Adds three regularisers and the epoch-level ensembling; no architecture change.
• Compared against: CLIP, CoOp, CoCoOp, ProDA and MaPLe tab. 1 §4.3.
• Related work the paper cites for weight-space ensembling as a motivation for its aggregation step §3.2.2.