Prompt-aligned Gradient for Prompt Tuning

In plain words

Large image-text models such as CLIP can label a photo by comparing it with a sentence like "a photo of a dog". Learning a few extra prompt vectors from labelled examples can make this more accurate. With only a handful of examples, though, it often goes wrong: the prompt fits those few photos so closely that the model ends up worse than with the untouched sentence.

ProGrad keeps the model's own opinion in the loop. At each training step it compares the update suggested by the new examples with the update that would keep the predictions close to the original model's. Where the two point against each other, the conflicting part is removed. The prompt then learns only what does not contradict what the model already knew. The change is small: switch it off and you get ordinary prompt tuning back.

The point is that when labelled data is scarce, protecting the pretrained knowledge can hold up better than trusting the few examples fully. The authors report this on image classification benchmarks.

TL;DR

ProGrad is CoOp-style prompt tuning for CLIP with one change: the cross-entropy gradient is projected so that it never works against the gradient of a KL term that keeps predictions near zero-shot CLIP §3.2. The author-reported ResNet-50 few-shot average beats CoOp at every shot count, by the most at 1 shot (62.61% vs 59.46%) tab. 1. Official PyTorch code is released under MIT, but its base-to-new context length, its λ and its zero-shot prompts differ from the paper in places (see Paper vs code). Relevant when few-shot prompt tuning falls below zero-shot.

Problem

CLIP's zero-shot classifier uses a hand-written prompt such as "a photo of a [CLASS]" §1. Prompt tuning replaces that text with learned context vectors, as in CoOp §1 §3.1. The authors find that such tuning can lose accuracy as training continues, and with one or two shots can fall below the zero-shot baseline §1 fig. 1. Grad-CAM maps show CoOp's attention drifting onto the background, while zero-shot CLIP stays on the foreground object §1 fig. 2. Existing remedies, early stopping and data augmentation, are not specific to prompts, which the authors see as the gap §1.

The method assumes that the zero-shot prediction is a useful teacher of general knowledge, and that disagreement between the labels and that teacher is what should be suppressed §3.2.

Method

1. Zero-shot teacher. Class probabilities come from the cosine similarity between the image feature and each class's text feature from a fixed template, scaled by CLIP's temperature §3.1 eq. 1. The teacher is computed without gradients prograd.py.
2. Student prompt. M learnable context vectors v are placed before each class-name embedding. Only v is trained; the image and text encoders are frozen §3.1 prograd.py. The cross-entropy against the labels is L_ce eq. 2.
3. Domain-specific direction. g_ce is the gradient of L_ce with respect to v §3.2.
4. General-knowledge direction. L_kl is a KL-type loss between the student and the teacher probabilities, weighted by the teacher. g_kl is its gradient with respect to v eq. 3 §3.2.
5. Projection. If g_ce and g_kl have a non-negative dot product, use g_ce unchanged. Otherwise subtract λ times the component of g_ce that lies along g_kl eq. 4 §3.2. λ = 1 removes that component fully; λ = 0 gives CoOp back §3.2.
6. Update. Take the optimizer step with the projected gradient. The paper does not name the optimizer in §3; the training recipe uses SGD app. A2.
7. Theory (sketch). The generalization bound combines the empirical risk, a domain-gap term, and Rademacher complexity on both domains eq. 6 §3.2. Simplifying assumptions are a large pretraining sample count and a small domain gap. Under them the paper argues that the estimation-error part of the CoOp bound is at least twice that of ProGrad eq. 8 §3.2. The paper does not measure the gap term.

# ProGrad update for the context vectors v  [§3.2] [eq. 2-4]
# Input:  images (B, 3, H, W); labels (B,); context vectors v
# Output: updated v

def prograd_step(images, labels, context, lr, lam):
    # Frozen zero-shot CLIP probabilities, no gradient  [§3.1] [eq. 1]
    teacher_probs = zero_shot_probs(images)            # (B, K)
    # Prompted CLIP probabilities for the current v  [§3.1] [eq. 2]
    student_probs = prompted_probs(images, context)    # (B, K)
    # Cross-entropy gradient: the domain-specific direction  [eq. 2]
    grad_ce = gradient(cross_entropy(student_probs, labels), context)
    # KL gradient: the general-knowledge direction  [eq. 3]
    grad_kl = gradient(kl(teacher_probs, student_probs), context)
    # Keep grad_ce unless it points against grad_kl  [eq. 4]
    if dot(grad_ce, grad_kl) < 0:
        # Remove the component of grad_ce that opposes grad_kl  [eq. 4]
        scale = dot(grad_ce, grad_kl) / norm_sq(grad_kl)
        grad_ce = grad_ce - lam * scale * grad_kl
    # SGD step on the adjusted gradient  [app. A2]
    context = context - lr * grad_ce
    return context

Key equations, as written in the paper. Here g_ce = ∇_v L_ce and g_kl = ∇_v L_kl §3.2.

p_\text{zs}(\bm{w}_i \mid \bm{x}) = \frac{\exp(\langle \bm{w}_i, \bm{f}\rangle/\tau)}{\sum_{j=1}^{K}\exp(\langle \bm{w}_j, \bm{f}\rangle/\tau)} eq. 1

\mathcal{L}_\text{ce}(\bm{v}) = -\sum_i \bm{y}_i \log p(\bm{t}_i \mid \bm{x}), \quad p(\bm{t}_i \mid \bm{x}) = \frac{\exp(\langle g(\bm{t}_i), \bm{f}\rangle/\tau)}{\sum_{j=1}^{K}\exp(\langle g(\bm{t}_j), \bm{f}\rangle/\tau)} eq. 2

\mathcal{L}_\text{kl}(\bm{v}) = -\sum_i p_\text{zs}(\bm{w}_i \mid \bm{x}) \log \frac{p(\bm{t}_i \mid \bm{x})}{p_\text{zs}(\bm{w}_i \mid \bm{x})} eq. 3

\tilde{g} = \begin{cases} g_\text{ce}, & \text{if } g_\text{ce}\cdot g_\text{kl} \ge 0 \\ g_\text{ce} - \lambda\,\dfrac{g_\text{ce}\cdot g_\text{kl}}{\lVert g_\text{kl}\rVert^2}\, g_\text{kl}, & \text{otherwise} \end{cases} eq. 4

Hyperparameters

Results

Author-reported. Setup: ResNet-50 unless stated; 11 datasets (ImageNet, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, EuroSAT, UCF101, DTD, SUN397); 3 seeds averaged; full test split §4. Baselines: zero-shot CLIP with one prompt (CLIP) and with prompt ensembling (CLIP++), linear probe (LP), CoOp, CoCoOp §4. ProGrad++ uses prompt ensembling to build the teacher §4. The per-dataset and standard-deviation tables are in the appendix and are not transcribed here.

Few-shot classification, average over 11 datasets, M = 16 tab. 1:

Robustness to distribution shift, trained on ImageNet, evaluated on ImageNet-V2, -Sketch, -A and -R tab. 2:

Base-to-new generalization, 4 shots, averaged over 11 datasets. H is the harmonic mean of base and new accuracy tab. 3 §4:

Comparison with knowledge distillation (LwF) and a cosine classifier, average over 11 datasets tab. 5 tab. 6:

λ ablation on DTD and OxfordPets, accuracy by shots (1, 2, 4, 8, 16) §4, λ table:

The paper reports a 3.2-point average gain over CoOp at 1 shot and gains of 5.1 to 9.5 points on FGVCAircraft, EuroSAT and Flowers102 at 1 shot; the unit of those per-dataset gains is not stated §4.

Paper vs code

• Context length for base-to-new. The paper uses M = 4 for base-to-new §4. The released base-to-new scripts set NCTX=16 base2new_train_prograd.sh base2new_test_prograd.sh. The release has no DG or cross-dataset script, so those settings cannot be rerun from it.
• λ at 16 shots. The paper sets λ = 0.8 at 16 shots §4. Every released script sets LAMBDA=1.0 prograd.sh, and the default is 1.0 train.py. No 16-shot override exists. The λ table has no 0.8 row §4, λ table.
• ProGrad++. The paper builds ProGrad++'s general direction from prompt ensembling §4. trainers/prograd.py builds its teacher from a single template prograd.py. Ensembling appears only in the zero-shot CLIP++ baseline zsclip.py. ProGrad++ cannot be rerun from this release as written.
• Zero-shot templates differ between trainers. The paper's prompt table lists "a type of aircraft, a photo of a {}" for FGVCAircraft and "a texture of {}" for DTD app. A3. trainers/prograd.py matches this prograd.py. trainers/zsclip.py uses "a photo of a {}, a type of aircraft." and "{} texture." zsclip.py. Which file produced the "CLIP" row is not stated, so the baseline depends on the trainer used.
• KL form. Eq. 3 is a KL-type expression eq. 3. The code computes the cross-entropy between teacher and student probabilities, multiplied by T² prograd.py. The two differ only by the teacher's entropy, which does not depend on v, so the gradients with respect to v are identical. The logged kl_loss is not a KL value.
• Projection granularity. Eq. 4 projects one vector eq. 4. The code runs the sign test and projection per parameter tensor trainer.py. Only one tensor is trained (the context vectors), so the two are the same here.
• Backbones. The robustness table has ResNet-101 and ViT rows tab. 2. The release has only RN50 configs .
• Consistent with the paper. Optimizer, initial learning rate, cosine schedule, warm-up at 1e-5, epochs by shot count, batch size 32, and initialisation from zeros plus the template embedding app. A2 match rn50.yaml and prograd.py. Also consistent: the projection formula eq. 4 and its λ = 0 limit.

Pitfalls

• The teacher sets the ceiling. The projection can only protect what the zero-shot prompt already gets right. The paper's failure analysis finds that many ProGrad failures are also misclassified by zero-shot CLIP [§4, failure cases] fig. 5. An oracle general direction gives a stronger model §4 fig. 7. A poor hand-written prompt therefore caps the method.
• Batch reduction is not stated. Eqs. 2 and 3 are per-sample sums; the code averages both losses over the batch prograd.py prograd.py. The projection is unchanged by scaling g_kl, so only the cross-entropy scale changes the step size. Changing the reduction changes the effective learning rate of the CE term.
• Cost. Each step runs a no-grad zero-shot forward pass and two backward passes, one for the KL term and one for the CE term prograd.py. The Dassl routine keeps the KL graph (retain_graph=True) and clones the KL gradients trainer.py. The paper calls the method efficient but gives no timing or memory figures §3.2.
• The temperature is CLIP's. The paper's τ is CLIP's learned temperature §3.1. The code uses CLIP's logit scale exponentiated prograd.py. It is not a separate hyperparameter.
• Initialisation template. The initialisation example in app. A2 is "a photo of a". The released code and app. A3 use a per-dataset template prograd.py. Use the per-dataset template, as the code does.
• Unstated optimizer details. The paper gives no momentum or weight decay. The code's Dassl defaults are listed under Hyperparameters.
• Precision. The code defaults to fp16 CLIP weights train.py. The paper does not state precision.
• Theory is a sketch. The bound eq. 6 and its approximation eq. 7 drop the pretraining estimation term and assume a small domain gap. The gap term is an integral probability metric that the paper does not estimate §3.2.
• The λ evidence is thin. The λ ablation covers only DTD and OxfordPets, and its effects are mixed: DTD at 16 shots is best at λ = 0.9, while OxfordPets at 16 shots is best at λ = 1.0 §4, λ table. The 16-shot λ = 0.8 setting in the main results is not covered by it.

Lineage

• Builds on: CoOp, the learned context-vector prompt method whose training recipe ProGrad reuses §3.1 app. A2. The code is built on CoOp and Dassl readme.md.
• Compared against: zero-shot CLIP and CLIP++, linear probe, CoOp, CoCoOp, learning without forgetting (LwF, knowledge distillation), and a cosine classifier §4. CLIP-Adapter results are in the appendix §4.
• Superseded by: not stated in the paper or the release.