MaPLe: Multi-modal Prompt Learning

In plain words

CLIP is a model with two halves: one reads pictures and the other reads text, and both map their inputs into a shared space where matching pictures and sentences sit close together. To use CLIP on a new task, people often add a few extra words to the text side, learned from examples, and leave the rest of the model alone. That is cheap, but it only changes one half of the model.

This paper adds learnable tokens to both halves. The text side gets prompt tokens, and the picture side gets matching tokens that are computed from the text ones through a small learned layer. The two halves are therefore tied together, so each can adjust to the other. The prompts are added not only at the input but also in several of the early processing layers of both halves.

The model is trained on a few labelled examples per class. Only the prompt parts are updated; the rest of the model stays frozen. On the paper's benchmarks this works better than text-only prompting, especially for classes the training data did not include.

TL;DR

• MaPLe learns prompt tokens in both the language and vision branches of a frozen CLIP ViT-B/16, with the vision prompts produced from the language prompts by a learned linear coupling function §3.2.3.
• Prompts are inserted in the first 9 transformer blocks for base-to-novel generalization §4.1.
• Author-reported: 78.55% harmonic mean (HM) over base and novel classes, averaged over 11 datasets, against 75.83% for Co-CoOp; novel-class accuracy rises from 71.69% to 75.14% tab. 3.
• Costs are small: about 3.55M trainable parameters, about 2.85% of CLIP, and inference is much faster than Co-CoOp at batch sizes above 1 tab. 6.
• The training code is public and matches the method in most respects; the differences are listed under Paper vs code.

Problem

CLIP-style vision-language models generalize to new concepts, but adapting them to a downstream task is hard. Full fine-tuning is costly and can damage the pre-trained alignment and overfit when labelled data is small §1. Earlier prompt-learning methods such as CoOp and Co-CoOp learn prompts only on the text side, so the image encoder is never adapted §1 §2. The authors argue this partial adaptation is sub-optimal, since the two encoders should both adjust to the task §1 §3.2. Co-CoOp conditions the prompt on each image, which lets it generalize to novel classes, but it has a high inference cost §4.6 tab. 6.

The authors' setting is few-shot: 16 labelled images per class, sampled at random §4.1.

Method

MaPLe builds on a frozen CLIP with a ViT-B/16 image encoder and a text encoder. The image encoder runs K transformer blocks over patch tokens and a class token; the text encoder runs K blocks over word tokens §3.1. Both feature vectors are projected into a common space, and class prediction uses the cosine similarity between the image feature and each class's text feature §3.1.

1. Text branch. Prompt tokens P_0 are placed after the start token in front of the class-name tokens. In the following blocks, new prompt tokens replace the prompt outputs of the block before §3.2.1 eq. 1 eq. 2.
2. Vision branch. The first vision prompt is appended after the patch tokens. In the next blocks, new vision prompts replace the previous ones, as on the text side §3.2.2.
3. Coupling. Every vision prompt is a linear function of the language prompt in the same layer, \tilde{P}_k = \mathcal{F}_k(P_k), mapping d_l=512 to d_v=768 §3.2.3 §4.1. The paper reads this as a bridge that lets gradients flow between the two branches §3.2.3.
4. Depth. Prompted layers run from the first layer up to depth J; for base-to-novel J=9, and the layers above J process the final prompts without new ones §3.2 §4.1.
5. Prediction. Image and text features are normalized and their cosine similarities are scaled by a temperature before the softmax over classes §3.1. Training minimizes cross-entropy on these logits, and only the prompt parts receive gradients while the rest of CLIP stays frozen §3 [code: trainers/maple.py#L191-L202] maple.py.

Equations as printed in the paper. Equations 1 to 3 are numbered in the paper; the others are unnumbered and cited by section.

Language prompts, layers i=1..J eq. 1:

[\ \underline{\hspace{0.3cm}}, \ W_{i}] = \mathcal{L}_{i}([P_{i-1}, W_{i-1}])

Language prompts, layers j=J+1..K and text output eq. 2 eq. 3:

[P_{j}, \ W_{j}] = \mathcal{L}_{j}([P_{j-1}, W_{j-1}])

z = \texttt{TextProj}(w_{K}^{N})

Vision prompts with the deep vision prompting scheme §3.2.2:

[c_i, E_i, \ \underline{\hspace{0.3cm}} \ ] = \mathcal{V}_{i}([c_{i-1}, E_{i-1}, \tilde{P}_{i-1}]) \quad i=1..J

[c_j, E_j, \tilde{P}_{j}] = \mathcal{V}_{j}([c_{j-1}, E_{j-1}, \tilde{P}_{j-1}]) \quad j=J+1..K

Coupling of vision prompts to language prompts §3.2.3:

\tilde{P}_k = \mathcal{F}_k(P_k)

Zero-shot classification as written in the paper §3.1. The paper's notation for \tau is discussed under Pitfalls:

p(\hat{y}|x) = \frac{\exp(sim(x, z_{\hat{y}})/\tau)}{\sum_{i=1}^{C}\exp(sim(x, z_{i}))}

Pseudocode for the forward pass, following the official code's behaviour. It covers the paper's design, not its typesetting. The batch is one labelled set of images and the class names are the training classes.

# MaPLe forward pass (training and inference) [§3.2] [eq. 1-3]
# Input: images (B, 3, 224, 224); class_tokens (C, 77) with the class names
# Output: logits (B, C); loss when labels are given

def maple_logits(images, class_tokens):
    # Prompt sets P_0 .. P_{J-1}, each (n_ctx, 512); P_0 is the input  [§3.2.1]
    language_prompts = [P[0], P[1], ..., P[J - 1]]
    # Vision prompts: a coupled copy of each language set  [§3.2.3]
    vision_prompts = [coupling[j](P[j]) for j in range(J)]  # (n_ctx, 768)
    # Patch and class tokens get positional embeddings first
    # [code: clip/model.py#L452]
    image_tokens = patch_embed(images) + positional_embedding  # (B, 1+M, 768)
    # Layer-1 vision prompts are appended after the image tokens  [§3.2.2]
    # [code: clip/model.py#L458]
    image_tokens = concat(image_tokens, vision_prompts[0])
    # Pre-norm runs after the prompts were appended  [code: clip/model.py#L463]
    image_tokens = ln_pre(image_tokens)
    # Layer-1 language prompts follow the start token  [§3.2.1]
    # [code: trainers/maple.py#L130-L131]
    text_tokens = insert_after_start(class_tokens, language_prompts[0])
    for layer in range(K):
        # Layers 2..J get new prompts; older prompt outputs are dropped
        # [§3.2.1] [§3.2.2]
        if 0 < layer < J:
            image_tokens = replace_vision_prompts(
                image_tokens, vision_prompts[layer]
            )
            text_tokens = replace_language_prompts(
                text_tokens, language_prompts[layer]
            )
        # Ordinary CLIP transformer block on each branch  [§3.1]
        image_tokens = image_block[layer](image_tokens)
        text_tokens = text_block[layer](text_tokens)
    # Read the class token and end token, project to the shared space  [§3.1]
    image_feature = image_proj(image_tokens[:, 0])    # (B, 512)
    text_feature = text_proj(end_token(text_tokens))  # (C, 512)
    # Cosine similarity, scaled by CLIP's learned logit scale
    # [code: trainers/maple.py#L197-L199]
    logits = exp(logit_scale) * (
        normalize(image_feature) @ normalize(text_feature).T
    )
    # Training: cross-entropy over the training classes
    # [code: trainers/maple.py#L202]
    return cross_entropy(logits, labels)

Hyperparameters

Base-to-novel and ImageNet-source settings differ where the table says so. Values in the code are the ones the authors' scripts run.

Parameter count, computed from the shapes in the trainer: language prompts 2×512, shallow coupling 512×768 + 768, eight deeper language prompt sets of 2×512, and eight deeper coupling layers of the same size. The total is about 3.55M, which matches the paper's 3.55M tab. 6.

Results

Author-reported. Setup: ViT-B/16 CLIP, 16 shots per class, 11 image recognition datasets, results averaged over 3 runs §4.1 tab. 3. The base-to-novel setting splits each dataset's classes into base and novel classes; the split protocol comes from Co-CoOp and is not described in this paper §4.1.

Base-to-novel generalization, average over 11 datasets tab. 3

Gains of MaPLe over Co-CoOp are +1.81 base, +3.45 novel and +2.72 HM tab. 3.

Base-to-novel generalization, per dataset tab. 3

Prompting design ablation, average over 11 datasets tab. 1

Comparison with CoOp trained in the Co-CoOp setting, average over 11 datasets tab. 2

Co-CoOp's training recipe lifts CoOp's novel accuracy by 6.8 points; CoOp† is the resulting baseline §4.3 app. A.

Cross-dataset evaluation (trained on all 1000 ImageNet classes with 16 shots, then tested on the other datasets; accuracy) tab. 4

Domain generalization (trained on ImageNet, tested on its variants; accuracy) tab. 5

Computational cost tab. 6 (FPS = frames per second, by batch size)

Training cost: the paper states MaPLe needs 5 epochs where Co-CoOp needs 10 §4.6.

Paper vs code

Compared with the official code at commit 69bce21ae8eda80ad6187534b2dce09cf6c59e17, a repository under the first author's GitHub account, linked from the abstract, and MIT-licensed. The discrepancies:

• Learning rate. Section 4.1 gives 0.0035 for the base-to-novel setting, which matches the code vit_b16_c2_ep5_batch4_2ctx.yaml. Appendix A states 0.0025 "for language and vision prompting" app. A; the code does not use 0.0025 for MaPLe.
• Learning rate, ImageNet source. Paper: 0.0026, 2 epochs, depth 3 §4.1; code: 0.0026, 2 epochs, depth 3 vit_b16_c2_ep5_batch4_2ctx_cross_datasets.yaml vit_b16_c2_ep5_batch4_2ctx_cross_datasets.yaml. These agree.
• Deep prompt sharing. Section 3.2.2 says sharing prompts across stages is better than independent prompts §3.2.2. The equations and the code do not share: each prompted layer has its own prompt tensor and its own coupling layer maple.py. Implement one tensor and one coupling layer per layer, as in the code.
• MaPLe†. The text and appendix describe MaPLe† with one coupling function for all layers §4.6 app. D tab. 6. The repository's MaPLe configs contain only the base-to-novel and cross-dataset setups, so MaPLe† is not in the released configs .
• Positional embeddings and norm. The paper places the vision prompts alongside the image tokens §3.2.2. The code appends them after the positional embeddings were added, so they get no positional embedding, and then applies the pre-norm to the whole sequence model.py model.py model.py.
• Precision. The paper does not discuss precision. The code runs in fp16 by default, creates the shallow coupling layer with .half(), and casts the prompts to fp16 inside the transformer blocks maple.py model.py model.py. Other precision modes still cast these prompts, which the paper does not discuss.
• Temperature. The paper's zero-shot equation puts \tau in the numerator only and does not define it §3.1. The code uses CLIP's learned logit scale maple.py maple.py.
• Checkpoint. Base-to-novel test loads the epoch-5 checkpoint [code: scripts/maple/base2new_test_maple.sh#L14]. The paper does not say which checkpoint it reports.
• Initialization of learnable language prompts. The paper says the first layer is initialized from the embeddings of "a photo of a " §4.1. The code sets only the first n_ctx = 2 tokens of that template as learnable (so "a photo"), and keeps "of a" and the class name as fixed tokens [code: trainers/maple.py#L84-L92] maple.py. This matches the appendix note on how fixed template tokens are used app. B.
• Training schedule. Cosine learning-rate decay, a 1-epoch constant warm-up at 1e-5, and fp16 are in the code only vit_b16_c2_ep5_batch4_2ctx.yaml vit_b16_c2_ep5_batch4_2ctx.yaml.

No further differences found in the prompt, coupling, depth and loss code.

Pitfalls

• Conflicting learning rates. Section 4.1 and the code say 0.0035; appendix A says 0.0025 §4.1 app. A vit_b16_c2_ep5_batch4_2ctx.yaml. The code is the reference here.
• Unstated optimizer details. The paper gives SGD and a learning rate but no momentum, weight decay or schedule. The schedule in the code is cosine with warm-up vit_b16_c2_ep5_batch4_2ctx.yaml. Momentum and weight decay are not set in this repository; they come from the Dassl training library that the install guide asks you to install separately INSTALL.md. Check Dassl's defaults before comparing numbers.
• Zero-shot equation. The paper's equation for class prediction puts \tau only in the numerator and never defines it §3.1. Use CLIP's logit scale, as the code does.
• Count mismatch in the text. Section 4.3 says MaPLe improves novel accuracy over CLIP on 6 of 11 datasets §4.3. Counted from table 3, MaPLe is higher than CLIP on 7 of 11 novel accuracies (ImageNet, Caltech101, OxfordPets, Food101, SUN397, EuroSAT and UCF101) and lower on 4 (StanfordCars, Flowers102, FGVCAircraft, DTD). The paper's averages are consistent with table 3; only the 6/11 count differs.
• Deep prompts are random except layer 1. Deep prompts are initialized randomly app. B maple.py. The paper reports that sensitivity to random deep prompts grows with depth §4.6.
• Prompt length and depth trade off. Longer prompts keep base accuracy but reduce novel accuracy, which the paper reads as over-fitting §4.6 fig. 4. Depth 9 gives the best result in the paper's sweep §4.6 fig. 4.
• Few-shot, not zero-shot. The method trains on 16 shots per class §4.1. The zero-shot baseline is CLIP with hand-crafted prompts for each dataset §4.3.
• Compute at small batch sizes. At batch size 1, Co-CoOp runs faster than MaPLe (64.6 vs 60.2 FPS), so the claim that MaPLe has "no overhead" holds for larger batches tab. 6 §4.6.
• FLOPs claim. The text says MaPLe's FLOPs exceed CoOp's and Co-CoOp's by only 0.1% §4.6. The GFLOPS column in table 1 gives 167.0 for MaPLe and 166.8 for deep language prompting tab. 1; the CoOp and Co-CoOp GFLOPS values appear only in commented-out source, so the 0.1% figure cannot be checked from the compiled tables.
• Author-reported numbers. All results are the authors' and come from one ViT-B/16 backbone; the code's README reports the same MaPLe figures README.md. No independent reproduction is recorded in this card.
• Table and figure numbers are inferred from the order of the LaTeX source, not from a compiled PDF. Appendix tables are cited by appendix section because their numbering is not confirmed.

Lineage

• Builds on CLIP, which provides the two encoders, the projections and the logit scale §3.1 (arXiv:2103.00020).
• Builds on CoOp, which learns continuous prompt vectors on the text branch only §2 (arXiv:2109.01134).
• Builds on Co-CoOp, which conditions the text prompt on each image and is the main comparison §2 tab. 3 (arXiv:2203.05557).
• Related deep visual prompting in a ViT; the paper calls it uni-modal and names its own deep vision prompting as similar in design §2 §3.2.2 (Jia et al., "Visual Prompt Tuning", ECCV 2022, as cited in the paper).
• Compared against: CLIP zero-shot, CoOp, CoOp†, Co-CoOp, deep vision prompting, deep language prompting and independent V-L prompting tab. 1 tab. 2 tab. 3.
• Superseded by: not stated in this card; no later card on this wiki records a successor.