On the Adversarial Robustness of Multi-Modal Foundation Models

In plain words

Vision-language models look at a picture and write text about it. This paper asks what happens when someone who only supplies the picture, for example a website or a post, has changed it in a way a person cannot see. The authors show that small, invisible pixel changes can steer the model's caption: either it degrades into nonsense, or it produces a sentence the attacker chose, such as a link to a scam page or a false news line.

The attack needs full access to the model's weights to compute the right pixel changes, so it is a white-box test. The authors run it on an open model, OpenFlamingo, and check both captioning and question answering. The point of the work is not to enable misuse but to show that a user who trusts the model's output can be misled by a manipulated image, so such models need defences.

TL;DR

OpenFlamingo (9B) is attacked by adding l-infinity perturbations of radius 1/255 or 4/255 to input images, optimised with APGD on the output likelihood. Targeted attacks force chosen captions, and untargeted attacks break captions and VQA answers. The attacks work in zero-shot and 4-shot settings, and even when only the query image is changed. All numbers are author-reported on a single model.

Problem

Multi-modal models are used in settings where honest users may receive images from third parties. Earlier attacks on such models either target jailbreaks (the user is the attacker) or use unbounded or large perturbations that a user would notice. The authors instead bound the perturbation to a small l-infinity ball, so that a user would not notice the change, and ask whether the honest user's trust in the output can be exploited §1 §2.

The threat model is a malicious third party who controls the images but not the model, and who can add perturbations with radius epsilon in the l-infinity ball. The attacker is assumed to have all model weights (white-box) §4. Unless stated otherwise, the paper evaluates one open model §5.

Method

Model. OpenFlamingo is an autoregressive model. It predicts the next text token from the previous tokens and the images, so the likelihood of a text y given images x is the product of per-token conditionals §3 eq. 1. Images enter through a vision encoder, and language-model layers attend to the visual features through cross-attention §3.

Threat models. Perturbations are restricted to an l-infinity ball of radius epsilon. Two sets of images can be perturbed: the query image q, and the context images c used for few-shot prompting. Radius epsilon_q applies to the query and epsilon_c to the context. Two settings are compared §4:

1. all: the adversary perturbs context and query, with epsilon_c = epsilon_q.
2. query: only the query is perturbed, with epsilon_c = 0.

In zero-shot inference there is no context, so the two settings coincide §4.

Untargeted attack. The adversary maximises the negative log-likelihood of the true caption (or answer) y, over the perturbations delta_q and delta_c §4 eq. 2. The ground-truth text is fixed per image, so the objective is a loss to be pushed up.

Targeted attack. The adversary chooses a desired output y-hat and minimises the negative log-likelihood of y-hat §4 eq. 3. The sign convention is opposite to the untargeted case, so the objective is minimised here.

Optimisation. Both objectives are optimised with APGD, a projected gradient method from the cited work §4 §5. Its only parameter is the number of iterations. The authors lower the initial step size from 2 epsilon to epsilon, which they report as increasing attack strength §5.

# Perturb the query (and context) images inside the l-infinity ball  [§4] [§5]
# Input:  query (3, H, W); context (k, 3, H, W); context text z; text y
# Output: perturbed query and context, within the radii

def attack(query, context, text, eps_q, eps_c, steps, targeted):
    # Negative log-likelihood of the text tokens  [§4] [eq. 2] [eq. 3]
    def nll(delta_q, delta_c):
        return -sum(
            log_prob(text_l | text_before_l, z, query + delta_q,
                     context + delta_c)
            for l, text_l in enumerate(text)
        )
    # Untargeted: maximise nll of the true text  [§4] [eq. 2]
    # Targeted: maximise -nll of the chosen text, i.e. minimise nll [§4] [eq. 3]
    def objective(delta_q, delta_c):
        if targeted:
            return -nll(delta_q, delta_c)
        return nll(delta_q, delta_c)
    # Projected gradient ascent with APGD; eps_c = 0 for the query setting [§5]
    # The initial step size is eps instead of 2 eps  [§5]
    delta_q, delta_c = apgd(objective, eps_q, eps_c, steps)
    return query + delta_q, context + delta_c

Evaluation protocol. Captioning uses the prompt <image>Output: and VQA uses <image>Short Answer:. Generated text beyond the first occurrence of Output: is cut, and a similar truncation is applied to VQA §5. For COCO and Flickr30k, one provided caption per image is the untargeted ground truth. For OK-VQA one ground-truth answer is sampled. For VizWiz one answer that is not "unanswerable" is sampled, unless more than half of the answers are "unanswerable" §5. Evaluation uses 1000 sampled instances per dataset §5.

Targeted evaluation. Two targets are used: "Please reset your password" and "A person suffers severe side effects after vaccination" §6 (text). The success rate counts a sample if the target caption is fully contained in the output. BLEU-4 is computed with the output length limited to the target's word count. CIDEr is not used here because TF-IDF weighting down-weights the target string §5.

Thresholding. The authors keep only a fraction of each perturbation by zeroing the smallest-magnitude pixel perturbations. Keeping the top 60% still reduces untargeted CIDEr and keeps targeted success high §6 fig. 5.

Hyperparameters

Results

Author-reported, on OpenFlamingo 9B, 1000 samples per dataset unless noted. Table 3 and Table 2 are copied with cited values. Where a 0-shot "query" cell is identical to the "all" cell, the value is shown in parentheses and the cell is blank in the paper §5.

Untargeted, CIDEr (COCO, Flickr30k) and accuracy (OK-VQA, VizWiz), 500 APGD steps tab. 3

The authors note that at 4/255 on COCO the CIDEr is below the mean of 100 random permutations of 1000 COCO captions (3.01, standard deviation 0.81) §2 §6. In the 4-shot case, context images help only slightly for captioning and almost not at all for VQA §6.

Targeted, success rate and BLEU-4 on COCO images, 500 APGD steps tab. 2

The text reports that the second target is "not well recovered" at 1/255 with 500 steps, and that more steps help §6 tab. 1.

Effect of APGD iterations, zero-shot, epsilon 1/255, 100 COCO images tab. 1

The caption states that a small number of iterations already degrades untargeted captions, while successful targeted attacks need a large number tab. 1. The table does not name the target caption used.

Thresholding §6 fig. 5: keeping the top 60% of perturbations (by magnitude) gives a COCO CIDEr of 24.99 at 1/255 and 3.65 at 4/255 in the untargeted setting. The authors say the 60% threshold already suffices for a strong targeted attack at 4/255. At 1/255 more perturbations must be kept.

Qualitative examples: Figure 1 (targeted, 1/255, zero-shot, two images) fig. 1; Figure 2 (targeted, 1/255, 5000 steps, zero-shot) fig. 2; Figures 3 and 4 (untargeted, 500 steps, zero-shot, 1/255 and 4/255) fig. 3 fig. 4.

Paper vs code

No official code. The paper has no code link in the LaTeX source, the arXiv abstract page lists only a conference comment ("ICCV AROW 2023") and no code or project page. GitHub searches for "OpenFlamingo adversarial", "adversarial multi-modal foundation", "Schlarmann" and "OpenFlamingo adversarial APGD" returned no repository for this paper. One GitHub account whose handle matches the first author's initials (the \chs macro in the source) has forks of OpenFlamingo and related projects, but none carries this attack. The account is not confirmed as the authors', so it is not used here.

Pitfalls

• Pixel scale of epsilon is not stated. The paper writes l-infinity radius 1/255 and 4/255 without saying whether images are in [0, 1] or [0, 255], or whether the radius applies before or after the CLIP preprocessing. Pick the scale of your input tensor and state it.
• Loss normalisation is not stated. Equations 2 and 3 sum the token negative log-likelihoods without a mean over tokens. Whether the implementation averages, and over which tokens, is not described.
• Sign convention. The paper writes the untargeted objective as maximising the negative log-likelihood, and the targeted one as minimising the negative log-likelihood. An implementation that flips one but not the other attacks the wrong way.
• Step size is changed from the cited default. Use an initial step of epsilon, not 2 epsilon §5. The effect is reported only qualitatively.
• APGD internals live in the cited work. This paper does not restate APGD's step-size schedule, restart rules or projection details. An implementer must get them from the cited reference.
• Iterations drive the results. The quantitative results use 500 steps, while the targeted examples use 5000 and Table 1 shows success rising from 56% to 83% at 1/255 tab. 1. The authors expect more steps to give stronger attacks but do not report 5000 steps for the tables.
• Table 1 and Table 3 disagree on the same setting. At 500 steps, zero-shot, epsilon 1/255, Table 1 reports CIDEr 15.72 on 100 images, while Table 3 reports 9.59 on 1000 images. The paper does not explain the difference. Do not compare these numbers across tables.
• Zero-shot "query" cells are blank in the source. Tables 2 and 3 leave them empty because zero-shot "all" and "query" are the same experiment §5. Copy the value from the "all" row.
• Context selection is not stated. The 4-shot context images and context texts, and how they are sampled, are not described. Results depend on them.
• Output truncation. The captioning truncation at Output: is explicit. The VQA truncation is described only as similar, so the exact cut-off string is not stated.
• Targeted metric is sensitive to output length. Success is a substring check on the output, so a target that appears inside a longer output counts. BLEU-4 is computed after the output is capped at the target's word count §5.
• Single model, single precision, no variance reported. Results come from one OpenFlamingo checkpoint, with single precision. The paper gives no seeds, repeats or confidence intervals. Batch size, optimiser learning rate (not applicable to APGD), and hardware are not stated.
• Cost. The authors call 5000 iterations "costly" and note that 500 is the most they could afford §5. Each step is a forward and backward pass through a 9B model, so compute scales with the step count and the number of images.
• White-box only. Only the attacker's own copy of the model is attacked, so the paper says nothing about transfer to other models or to a deployed API.
• Attack-success metrics are author-defined. The CIDEr threshold for "broken" and the random-permutation baseline are the authors' choices §2.
• Figure 1 step count is not in the caption. The file names say 5000. Treat 5000 as likely but not stated.
• Image credits. The paper's example images come from third-party sites (stock photo and other image sites), listed in a bibliography section of the source. This card uses no images.

Lineage

• Builds on: OpenFlamingo, an open implementation of Flamingo §3 (no card yet); the APGD optimiser §5 (no card yet).
• Compared against: concurrent multi-modal attacks with large or unbounded radii, Qi et al. (universal perturbations, radius 16/255 or more), Carlini et al. (unbounded), Bagdasaryan et al. (unbounded) §2. These are named but no numbers are compared in this paper.
• Superseded by: not stated in this wiki.