PEST: Parameter Efficient Steering of Blackbox VLMs via Agentic Few-shot Alignment for Hateful Meme Moderation

In plain words

Hateful memes are hard to moderate because their meaning comes from the picture and the text together. Most earlier work only decides whether a meme is hateful. Explaining why, or suggesting a response a user could accept, has mostly been studied on its own. This paper asks for all three at once: a label, a short explanation, and, for hateful memes, an intervention.

Labeled data for that combined task does not exist, and it is expensive to make. The authors therefore use small helper models. Each one, a 3-billion-parameter model, is trained for one job: captioning, explaining, or proposing an intervention. The helpers write extra text for a handful of example memes. A large closed vision-language model then sees those enriched examples in its prompt and does the classification, explanation and intervention itself. Its weights are never changed.

At test time only the captioning helper runs. The explanations and interventions for the examples are written in advance. The idea is to steer a large model with a small one, without training the large one.

TL;DR

• Four PaliGemma-3B helpers are fine-tuned: caption (A_C), label-aware explanation (A_E), common-sense reasoning, and intervention (A_I) §5.1 tab. 2.
• For a test meme, only A_C runs. The top 2, 4 or 8 support memes by SigLIP image similarity are added to a few-shot prompt with their precomputed text §5.2.
• A frozen VLM answers with a label, an explanation, and an intervention if the label is hateful §5.2 app. Prompts and definitions.
• Author-reported 8-shot GPT-4o: 80.26 accuracy and 80.25 macro-F1 on FHM; 89.10 and 89.07 on MAMI tab. 3.
• The official code is MIT-licensed. It has the fine-tuning and open-model few-shot scripts. It has no GPT-4o inference code, and several settings differ from the paper (see Paper vs code).

Problem

• Classification-only moderation leaves out explanation and intervention. Earlier work studies these pieces separately §1.
• Earlier explanation and intervention work assumes the label is already known. Real moderation cannot assume that §1.
• No dataset holds labels, explanations and interventions together. Annotating one is costly and subjective §1 §3.
• Fine-tuning large VLMs needs data and compute. Few-shot prompting avoids fine-tuning, but it had not been used for classification, explanation and intervention together §1 §2.

Method

Overview

The method has an offline stage and an online stage.

• Offline: fine-tune the task agents, then run them over the support memes to produce captions, explanations and interventions §5.1 §5.2 fig. 2.
• Online: retrieve similar support memes for each test meme and prompt a frozen VLM with them §5.2.

Agents, all fine-tuned from google/paligemma-3b-pt-448 §5.1:

• A_C, caption agent: writes a meme-oriented caption for an input meme §5.1.
• A_E, label-aware explanation agent: writes an explanation given the meme's ground-truth label §5.1.
• Common-sense step and A_I, intervention agent: a common-sense reasoning model is trained first. The intervention model then writes an intervention using that reasoning §5.1 §5.2.

Agent training data §4 tab. 1:

• MemeCap train and validation, 5,828 memes, for captions §4.
• HateRed-Aug, 8,370 memes, for explanations: 2,982 hateful and 5,388 non-hateful. The non-hateful explanations are new annotations by the authors §3 §4.
• MemeSense, 299 training samples, for common sense and for interventions §4 tab. 1.

Support and evaluation sets §4 tab. 1:

• FHM support: 8,370 memes, the FHM seed of HateRed-Aug. Their explanations come from HateRed-Aug §5.2.
• MAMI support: 10,000 training memes, 5,000 misogynistic and 5,000 not §4.
• Evaluation: FHM test_seen extended with explanations and interventions, 983 memes (485 hateful, 498 non-hateful). MAMI test extended, 945 memes (478 misogynistic, 467 non-misogynistic) §4 tab. 1.

Exemplar selection and enrichment §5.2: each exemplar is chosen by cosine similarity of SigLIP image embeddings to the test image. Exemplars are enriched with the caption (all), the explanation (all), and the intervention (hateful only). The paper does not state the order in which exemplars appear in the prompt.

Agent training, following §5.1 and §5.3:

# Fine-tune one task agent from PaliGemma-3B  [§5.1] [§5.3]
# Input:  (image, prompt, target text) examples for one task
# Output: fine-tuned agent

def train_agent(examples):
    # Public 3B checkpoint, weights in bfloat16  [§5.3]
    model = load_pretrained("google/paligemma-3b-pt-448")
    # Paper: full fine-tuning. Code freezes the vision tower and projector.
    # [code: finetuning/paligemma_caption_generator_finetuning.py#L106-L109]
    freeze(model.vision_tower)
    freeze(model.multi_modal_projector)
    # Adam, lr 2e-5, weight decay 1e-6, batch size 2  [§5.3]
    for batch in batches(examples, size=2):
        loss = model.loss(batch.images, batch.prompts, batch.targets)
        update_weights(model, loss)
    return model

Enrichment of support memes, offline §5.2:

# Enrich support memes once, offline  [§5.2] [fig. 2]
# Input:  support memes with labels; agents A_C, A_E, A_I
# Output: per meme: caption, explanation, optional intervention

def enrich_support(support_memes, agents):
    enriched = []
    for meme in support_memes:
        # Caption for every support meme, from A_C  [§5.2]
        caption = agents.caption(meme.image)
        # FHM: explanation from HateRed-Aug; MAMI: from A_E  [§5.2]
        explanation = explanation_source(meme, agents)
        intervention = None
        # Intervention only for hateful memes  [§5.2]
        if meme.label == HATEFUL:
            # Common sense first, then the intervention  [§5.1]
            reasoning = agents.common_sense(meme.image)
            intervention = agents.intervene(meme.image, reasoning)
        enriched.append((meme, caption, explanation, intervention))
    return enriched

Inference on one test meme §5.2 §5.3 app. Prompts and definitions:

# Classify, explain and intervene on one test meme  [§5.2]
# Input:  test meme, enriched support set, shots n in {2, 4, 8}, VLM
# Output: label, explanation, intervention (only if hateful)

def pest_predict(test_meme, enriched_support, shots, vlm):
    # Only the caption agent runs at test time  [§5.2]
    caption = agents.caption(test_meme.image)
    # Most similar support memes by SigLIP image-embedding cosine  [§5.2]
    exemplars = top_k_by_cosine(test_meme.image, enriched_support, shots)
    # One user turn per exemplar (image, caption, OCR text), then the
    # assistant answer built from its enriched fields  [app. Prompts]
    messages = build_chat(exemplars, test_meme, caption)
    # Frozen VLM: temperature 0.001, at most 100 new tokens  [§5.3]
    reply = vlm.chat(messages, temperature=0.001, max_new_tokens=100)
    return parse_reply(reply)

No equation in the paper is numbered. The retrieval rule is stated in prose as cosine similarity between SigLIP embeddings §5.2.

Hyperparameters

Datasets

Copied from tab. 1 in the paper, condensed:

H is hateful or misogynistic, NH is non-hateful or non-misogynistic.

Results

Author-reported. Setup: FHM and MAMI test splits (extended with explanation and intervention annotations) §4; baselines as cited in §5.4; few-shot VLMs with 8, 4 or 2 shots §6. Hardware for inference is not stated beyond the deployment note in §8.2.

Agent quality tab. 2

Held-out test splits, using the four fine-tuned PaliGemma-3B agents. rgL = ROUGE-L, ss = semantic similarity, bsf1 = BERTScore-F1.

Classification tab. 3

Accuracy (acc) and macro-F1 (mF1), in percent. Baseline rows are as the paper lists them. The paper does not say whether they were re-run or copied.

The paper's own summary: 8-shot GPT-4o is best on classification accuracy among the listed methods on both datasets §6. The margins over fine-tuned baselines it reports on FHM mF1 are about 8% (PromptHate), 5.4% (ProCap) and 4.6% (M2KE) §6.

Explanation and intervention tab. 3

ROUGE-L (rgL), semantic similarity (ss) and BERTScore-F1 (bsf1). The paper's column "sprt" is undefined in the table caption beyond the word "support". Its unit is not stated.

Author-reported observations §6: GPT-4o gives the best explanations. The open models (InternVL, Pixtral) give the best interventions. Zero-shot VLMs are much worse than their 8-shot versions.

Toxicity tab. 4

Mean (standard deviation) PerspectiveAPI TOXICITY score, 8-shot setup §6.

Author-reported: no mean exceeds 0.3. Interventions are the least toxic.

Human evaluation §6

300 memes: 200 for explanations, 100 for interventions. Two experts rated semantic similarity on a 10-point scale, which the paper says is normalised to [0, 1] (the normalisation is not described). Mean scores per expert (expert 1, expert 2) in the paper's text §6:

• Explanations, GPT-4o: FHM (0.708, 0.72), MAMI (0.736, 0.756).
• Explanations, InternVL3.5-8B: FHM (0.654, 0.654), MAMI (0.672, 0.688).
• Interventions, GPT-4o: FHM (0.568, 0.592), MAMI (0.624, 0.608).
• Interventions, InternVL3.5-8B: FHM (0.672, 0.644), MAMI (0.748, 0.736).

Human evaluation covered GPT-4o and InternVL only. The paper says Pixtral followed almost the same pattern as InternVL §6.

Analysis §7 fig. 3 fig. 4

Author-reported textual analysis, by subgroup (correct or wrong, hateful or not):

• Token count: explanations are shorter than interventions in all groups. GPT-4o's lengths are the most consistent across groups fig. 3.
• Type-token ratio: explanations are more lexically varied than interventions. Interventions are longer but more repetitive fig. 3.
• Unigram perplexity: interventions have higher perplexity than explanations fig. 3.
• Sentiment (VADER): explanations for hateful predictions are more negative; GPT-4o interventions are more positive than the open models' fig. 4 §7.

Ablation §8.1

8-shot, FHM, GPT-4o. Variants: no agent (Vn), each single agent, chain-of-thought (CoT) and zero-shot (ZS) prompting. Classification mF1 drops are −4.36 for zero-shot and −3.55 for CoT. Relative drops in semantic similarity against the full three-agent setup (explanation, intervention): only A_C (29%, 28%); only A_E (0%, 28%); only A_I (42%, 3%); Vn (34%, 29%); CoT (19%, 25%); ZS (19%, 26%) §8.1.

Deployment cost §8.2

Fine-tuning each 3B agent on commodity hardware: 30 GB VRAM, batch size 1, bfloat16. Each agent needs about 10 GB at inference. Only A_C is deployed next to the black-box VLM. With 8-shot GPT-4o, one sample takes about 6–7 seconds, using about 1,350 input and 50 output tokens. The author estimate is about USD 0.0041 per sample §8.2.

Other embedding retrievers app. Other embedding retrievers

Table 5 in the paper gives 2-, 4- and 8-shot results with BLIP and CLIP retrievers. Only the 8-shot rows are reproduced here. The paper's conclusion: all three retrievers perform similarly, and SigLIP is the most consistent.

The 8-shot SigLIP rows are in tab. 3. The rows for 2 and 4 shots are not reproduced here, so read tab. 5 for them.

Significance app. Significance test tab. 6

Exact two-sided McNemar test on classification, p-values:

Only Pixtral's 8-shot versus its zero-shot version on MAMI is not significant at the paper's level app. Significance test.

Paper vs code

Compared against the official repository at c8c529994988b072ea575b75eef91a7f265281ba, MIT, copyright naquee-rizwan. The paper's introduction links it §1. Its README cites this paper and lists its authors.

1. Number of agents. The text describes three agents §5.1. Table 2 lists four fine-tuned agents, with common sense separate from intervention tab. 2. The code trains four models: caption, explanation, common sense and intervention paligemma_caption_generator_finetuning.py, paligemma_target_aware_interpretability_generator_finetuning.py, paligemma_common_sense_analysis.py, paligemma_intervention_generation.py. Implement four.
2. Full fine-tuning vs frozen parts. The paper says full fine-tuning §5.1 §5.3. All four scripts set requires_grad = False on the vision tower and the multi-modal projector: paligemma_caption_generator_finetuning.py, paligemma_target_aware_interpretability_generator_finetuning.py, paligemma_common_sense_analysis.py, paligemma_intervention_generation.py.
3. Epochs. The paper says 2 epochs for the small agents §5.3. The code uses 1 for captions paligemma_caption_generator_finetuning.py and explanations paligemma_target_aware_interpretability_generator_finetuning.py, and 2 for common sense paligemma_common_sense_analysis.py and intervention paligemma_intervention_generation.py.
4. Optimizer. The paper says Adam §5.3. The code sets no optimizer, but sets adam_beta2=0.999 in all four scripts paligemma_caption_generator_finetuning.py. Which optimizer that gives is not stated in the repo.
5. Warmup and batch size. Not in the paper. The code uses 2 warmup steps and per-device batch 2 in all four scripts paligemma_caption_generator_finetuning.py, paligemma_caption_generator_finetuning.py.
6. Inference temperature. The paper calls 0.001 "very low" to ensure reproducibility §5.3. The code samples: do_sample=True with temperature 0.001 fhm.py, fhm.py. Pixtral uses vLLM sampling at temperature 0.001 fhm.py. Repeat runs can differ.
7. Agent max new tokens. The paper says 100 for all inference §5.3. Agent inference in code uses 50 paligemma_caption_generator_finetuning.py, paligemma_target_aware_interpretability_generator_finetuning.py, paligemma_common_sense_analysis.py, paligemma_intervention_generation.py. The few-shot VLMs use 100, as the paper says fhm.py.
8. Explanation exemplars for FHM. The paper says FHM exemplars use HateRed-Aug explanations and that A_E is used only for MAMI §5.2. The code's FHM script builds its explanation lookup from one CSV named for non-hateful FHM training memes fhm.py. A missing key returns an empty string fhm.py. The CSV is not in the repository, so whether hateful exemplars are covered cannot be checked. The MAMI script loads explanations from the explanation agent's JSON output mami.py, which matches the paper.
9. Intervention for exemplars. The code attaches an intervention only when the exemplar's label is hateful and an intervention exists fhm.py. This matches the paper §5.2.
10. Explanation prompt in the intervention and common-sense inference scripts. Their test loops build the explanation prompt ("Explain why this meme is harmful"), not an intervention prompt paligemma_intervention_generation.py, paligemma_common_sense_analysis.py. The training prompt for intervention is different paligemma_intervention_generation.py. Use the training prompts when reproducing.
11. Explanation prompt words. Training uses "hateful" and "not-hateful" paligemma_target_aware_interpretability_generator_finetuning.py. Inference in the same script uses "harmful" and "not-harmful" paligemma_target_aware_interpretability_generator_finetuning.py.
12. Verification filter. The explanation dataset drops non-hateful rows marked -1 in a verification column paligemma_target_aware_interpretability_generator_finetuning.py. The paper does not mention this filter. Its counts (5,388 non-hateful) may therefore differ from what the code trains on.
13. Evaluation sets for agents. The paper evaluates agents on held-out splits of MemeCap, HateRed and MemeSense with n = 559, 246 and 134 §6. The explanation and intervention scripts read HARM_P_test_explanations.csv paligemma_target_aware_interpretability_generator_finetuning.py. That file name is not used in the paper, so whether it is the same split cannot be verified.
14. GPT-4o. The paper's main GPT-4o results were run through Azure §5.3. The repository has no GPT-4o inference code. Its only mention is commented-out calls with the checkpoint name gpt-4o-hate-alert fhm.py. The GPT-4o numbers cannot be reproduced from the repository.
15. Retriever for the Pixtral FHM run. The paper reports SigLIP results for the main table §5.2. The Pixtral FHM script's main loop runs the CLIP encoder fhm.py. The Pixtral MAMI script runs SigLIP with 8 shots only mami.py, mami.py. Which retriever produced the reported Pixtral FHM numbers is not verifiable.
16. Few-shot prompt wording. The paper's appendix prompt uses "non-hateful" app. Prompts and definitions. The code's system message uses "not-hateful" and inlines the definitions fhm.py. Labels in the answer are the same.
17. Exemplar order. The code returns exemplars with the most similar last rices.py. The paper does not state an order.
18. Shot order in MAMI. The MAMI InternVL run loops over 8, 4, 2 shots mami.py. The paper reports 2, 4 and 8. This affects only run order.

No discrepancies were found for the cosine retrieval itself rices.py or for the top-n selection rices.py.

Pitfalls

• Four agents, one captioner at test time. Implement four fine-tuned models. At inference, only the caption agent runs §5.2. Explanations and interventions for support memes must be computed offline first.
• Label leakage risk in A_E. A_E conditions on the ground-truth label §5.1. It is used only to build silver data for support memes. Running it on test memes would leak the label.
• Intervention is conditional. It is written only for hateful exemplars §5.2. At test time it is asked for only when the model predicts hateful §5.2.
• Sampling at 0.001. Generation is sampled, not greedy, in both code paths fhm.py, fhm.py. Expect small run-to-run differences. Fix seeds if exact reproduction matters.
• GPT-4o version unknown. The paper names GPT-4o via Azure §5.3 but does not state a model snapshot or date. Results may not transfer to later GPT-4o versions.
• Metric backbones. Semantic similarity uses all-MiniLM-L6-v2 §5.3. BERTScore-F1 cites Zhang et al. (2020) but does not name the backbone model. PerspectiveAPI's version is not stated §6.
• Human-rating scale. The paper describes a 10-point scale, then says the scores are normalised to [0, 1] without describing how §6.
• Annotation quality. Pilot accuracy above 95% (judged by the first author). Two-annotator re-check on all samples: over 94% agreement, Cohen's κ = 0.932, about 2% of samples rewritten §3.2. Annotators were six researchers aged 21–30 §3.1.
• Undefined column. "sprt" appears in tab. 3 and its unit is not defined in the paper. Do not treat it as a score.
• Baseline provenance. The paper does not say whether its baseline numbers were re-run on its splits or copied from the cited works §5.4 tab. 3. Compare with care.
• Dataset licence not stated. The paper and repository point to new annotations and a dataset on Hugging Face (linked from the repository README). Neither states a licence for the new annotations. This card uses no dataset content. The paper says the dataset will be released after acceptance [Ethics statement].
• Language. The dataset is primarily English [Limitations]. The authors say helper agents would need checking in other languages.
• Cost depends on the prompt. The per-sample cost estimate assumes about 1,350 input tokens with 8 shots §8.2. It changes with shot count and image size.
• Statistical margin. Pixtral's 8-shot gain over zero-shot on MAMI is not significant (p = 0.404) tab. 6.

Lineage

• Builds on: MemeCap (captions), HateRed (explanations; its dataset is extended to HateRed-Aug), and MemeSense (common sense and interventions) §2 §4. No arXiv ids are given in this card; the paper names them only by author and year.
• Compared against: PromptHate, ProCap, ModHate, Few-Shot, U-CoT+, M2KE, Vlm-Lim and LoReHM for classification §5.4; HateRed for explanations and MemeSense for interventions §5.4.
• Superseded by: not known from content in this wiki.