Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs

In plain words

CLIP-style models put pictures and sentences into one shared space, so that a picture and its caption land close together. They are weak with long text, and they read the image and the text separately, which makes it hard for them to tell apart captions that use the same words in a different structure.

This paper starts from a multimodal large language model, a model that already understands images and text, and turns it into such an embedder. It does this in two steps. First, it trains only the language part, using text alone, so that its sentence vectors copy the similarity pattern of a strong text-embedding model. Second, it trains on matching image-text pairs with a contrastive loss. Before that loss runs, it removes candidates that look too similar to the true match, since those are probably correct too. It then makes the model separate the true match from the hardest of the remaining wrong candidates. The goal is one model that handles many retrieval and matching tasks.

TL;DR

UniME is a two-stage recipe for turning a multimodal LLM into a general-purpose image-text embedder. Stage 1 distils batch-level similarity structure from NV-Embed V2 into the LLM using text only. Stage 2 fine-tunes with InfoNCE over the positive and k=8 hard negatives, after removing false negatives. The authors report the best numbers on MMEB with LLaVA-OneVision-7B, and gains over E5-V and VLM2Vec with the same base models. Relevant if you need one embedding model for retrieval, classification-as-retrieval and compositional matching, and can afford LoRA fine-tuning on 8 A100s.

Problem

• CLIP limits text to 77 tokens, which hurts long-caption retrieval §1. Its dual encoders process images and text separately, which hurts instruction- following multimodal retrieval §1. It also shows bag-of-words behaviour on compositional queries §1.
• MLLMs are trained with autoregressive next-token prediction, which limits their representation quality compared with contrastive models §1. E5-V adapts an MLLM with unimodal contrastive learning on sentence pairs, and the authors name two constraints: limited data scale and diversity, and the causal attention mask §1.
• In standard InfoNCE training, false negatives in a batch make it hard to separate true hard negatives from wrong pairs that look like positives §1 §3.3.
• Assumptions: images are fed at 336×336 in stage 2 §4.1. Stage 1 needs a teacher embedding model and offline text embeddings §3.2 §4.1.

Method

The model is an MLLM with three parts: a vision tower, a projection layer and an LLM backbone §3.1. Queries and candidates are embedded by the same network, each with a task-specific prompt. Candidates are ranked by similarity §3.1 eq. 1:

\boldsymbol{P} = \text{Rank}(\Theta(\phi(\boldsymbol{q}), \phi(\boldsymbol{c}))) eq. 1

where \phi is the MLLM embedder and \Theta the similarity. Inputs may be unimodal or interleaved image-text §3.1. The embedding is the hidden state of the last token §3.2, normalised for the distillation and tuning losses §3.2 §3.3.

Stage 1: textual discriminative knowledge distillation §3.2.

1. Decouple the LLM from the MLLM and feed it text only, with the E5-V embedding prompt " Summary above sentences in one word:" §3.2 §3.1.
2. Compute the student embeddings e_s (LLM, trainable) and the teacher embeddings e_t (NV-Embed V2, computed offline) for a batch of n sentences §3.2.
3. Compare the batch-wise similarity distributions of student and teacher with a KL divergence, summed over the batch, at temperature \tau §3.2 eq. 2.
4. Only the LLM is trained, with LoRA; the other MLLM parts stay frozen §3.2 §3.4.

\mathcal{L}_{KL} = \sum_{i=1}^{n}\mathrm{KL}\left( \frac{\exp(e_{s_i}^\top e_{s_i} / \tau)}{\sum_{j=1}^{n}\exp(e_{s_j}^\top e_{s_i} / \tau)} \,\Big\Vert\, \frac{\exp(e_{t_i}^\top e_{t_i} / \tau)}{\sum_{j=1}^{n}\exp(e_{t_j}^\top e_{t_i} / \tau)} \right) eq. 2

As printed, the numerator of eq. 2 is a self-similarity term. See Pitfalls.

Inference for stage 1 §3.2. Restore the vision encoder and projector. Each unimodal input uses its modality prompt. For an interleaved input, each part is encoded separately and the vectors are summed element-wise.

Stage 2: hard negative enhanced instruction tuning §3.3. Stage 2 starts from the stage-1 model and trains on image-text pairs from MMEB §3.4 §4.2.

1. False-negative filter. For query i, the threshold is \alpha = \cos(e_q, e_c^{+}) + \beta. Every other candidate in the batch whose cosine similarity to the query is above \alpha is excluded §3.3.
2. Hard negative sampling. Among the remaining candidates, take the k with the highest cosine similarity. If fewer than k remain, duplicate existing hard negatives to keep size k §3.3 eq. 3.
3. Loss. InfoNCE over the positive and the k hard negatives, with temperature \tau §3.3 eq. 4.

e^{-}_{c} = \text{Rank}_{k}\left(\cos(e_{q}, e_{c})\right), \quad \text{where } e_{c} \notin \{e^{+}_{c}, e^{*}_{c}\} eq. 3

Here e^{*}_{c} are the filtered false negatives, e^{+}_{c} the positives, and \text{Rank}_k returns the top k scores §3.3.

\mathcal{L} = -\log \frac{\exp(\cos(e_{q}, e_{c}^{+})/\tau)} {\exp(\cos(e_{q}, e_{c}^{+})/\tau) + \sum_{i=1}^{k}\exp(\cos(e_{q}, e_{c_{i}^{-}})/\tau)} eq. 4

The paper's own notation calls the set of hard negatives k as well §3.3.

Training recipe §3.4. Stage 1 uses QLoRA on the LLM component. The text says the trainable part is typically under 5% of all parameters §3.4. Stage 2 uses gradient caching (GradCache, following VLM2Vec) to decouple the contrastive loss from encoder backpropagation, and QLoRA for parameter-efficient fine-tuning of the MLLM §3.4. Stage-2 LoRA rank and other settings are in the Hyperparameters table.

Algorithm sketch. The paper gives no algorithm block. The pseudocode below is our reading of §3.2, §3.3 and eq. 2-eq. 4. It is not the authors' code. The official loss code covers stage 2 only loss.py.

# Stage 1: textual discriminative knowledge distillation  [§3.2] [eq. 2]
# Input: n sentences; teacher embeddings (n, d), computed offline
# Output: scalar KL loss over the batch similarity distributions

def distill_loss(sentences, teacher_embeddings, tau):
    # Student: text-only pass through the LLM with the embedding prompt
    student_embeddings = normalize(embed_last_token(sentences))  # (n, d)
    # Batch similarity matrices; row i = similarities of sample i  [eq. 2]
    student_sims = student_embeddings @ student_embeddings.T  # (n, n)
    teacher_sims = teacher_embeddings @ teacher_embeddings.T  # (n, n)
    # Temperature-scaled softmax over each row  [eq. 2]
    student_probs = softmax(student_sims / tau, axis=1)
    teacher_probs = softmax(teacher_sims / tau, axis=1)
    # KL(student || teacher), summed over samples  [eq. 2]
    return sum(kl_divergence(student_probs, teacher_probs))

# Stage 2: hard negative enhanced instruction tuning  [§3.3] [eq. 3] [eq. 4]
# Input: n (query, positive) pairs in one batch; k, beta, tau
# Output: scalar InfoNCE loss

def tuning_loss(queries, positives, k, beta, tau):
    # Embed both sides with their task prompts, L2-normalised  [§3.1]
    query_embeddings = normalize(embed(queries))       # (n, d)
    positive_embeddings = normalize(embed(positives))  # (n, d)
    sims = query_embeddings @ positive_embeddings.T    # (n, n), cosine
    # Per-query threshold alpha = positive similarity + beta  [§3.3]
    thresholds = diagonal(sims)[:, None] + beta
    # Drop false negatives and the positive itself from the pool  [§3.3]
    pool = where(sims > thresholds, -inf, sims)
    fill_diagonal(pool, -inf)
    # The k best remaining candidates are the hard negatives  [eq. 3]
    hard_idx = topk_indices(pool, k)                   # (n, k)
    hard_negatives = positive_embeddings[hard_idx]     # (n, k, d)
    # Score the positive first, then the k hard negatives  [eq. 4]
    negative_scores = sum(
        query_embeddings[:, None] * hard_negatives, axis=-1
    )                                                  # (n, k)
    scores = concat([diagonal(sims)[:, None], negative_scores], axis=1)
    # Cross-entropy with the positive at index 0, scaled by tau  [eq. 4]
    return mean(cross_entropy(scores / tau, labels=0))

Figures are described in words only. Figure 1 is the framework teaser §1. Figure 2 shows the stage-2 framework, with false-negative filtering and hard negative sampling §3.3 fig. 2.

Hyperparameters

Values are as stated in the paper unless marked as taken from code. Table 6 tab. 6 gives the full training-setup table for Phi-3.5-V and LLaVA-1.6.

Results

Author-reported. Setup: MMEB (36 datasets: 20 in-distribution, 16 out-of-distribution) with Precision@1 §4.2 tab. 1; zero-shot cross-modal retrieval on Flickr30K, COCO, ShareGPT4V, Urban1K and SugarCrepe with Recall@1 §4.2 app. A.2 tab. 2. Baselines are CLIP-family models, E5-V and VLM2Vec. Baseline MMEB numbers for CLIP, SigLIP, BLIP-2 and MagicLens come from VLM2Vec; E5-V, VLM2Vec and UniME were re-run by the authors app. B.2. Table and figure numbers below follow the LaTeX source order. The compiled PDF was not read. The per-dataset MMEB table tab. 9 is omitted. The colour-coded deltas in the source tables are omitted; they are differences of the numbers shown.

Headline numbers. With Phi-3.5-V, UniME stage 1 alone gives a 4.2% average gain over E5-V; with LLaVA-1.6 the gain is 4.1% §4.3 tab. 1. After stage 2, UniME (Phi-3.5-V) is 1.3 points above VLM2Vec (64.2 vs 62.9) tab. 1. The text also claims 10.3% over VLM2Vec with LLaVA-1.6 §4.3, but Table 1 gives 66.6 vs 63.3, a 3.3-point gap (see Pitfalls). The largest reported MMEB result is UniME with LLaVA-OneVision-7B, at 70.7 overall tab. 1.

MMEB, Precision@1 tab. 1. Columns: meta-task scores, then IND, OOD and Overall averages.

† UniME with stage 1 only. ‡ UniME with stage 1 and stage 2 tab. 1. The first block is zero-shot; the rest is fine-tuned on MMEB tab. 1.

Retrieval, Recall@1 tab. 2. Columns: text→image (t→i) and image→text (i→t) per dataset. SugarCrepe has three categories.

Stage ablation, Phi-3.5-V tab. 3. Averages over MMEB, short-caption retrieval, long-caption retrieval and compositional retrieval.

False-negative margin β, Phi-3.5-V tab. 4. FalseNeg% is the share of samples in which a false negative was filtered out. The paper reports β=0.1 as best overall on its MMEB average §5 tab. 4.

Number of hard negatives k, Phi-3.5-V tab. 5. HardNeg% is k as a share of the batch, so the batch size behind these numbers is about 1024 (derived: 8/1024 = 0.8%).

LoRA rank, Phi-3.5-V tab. 8. Stage 1 is best at rank 32 and stage 2 at rank 16, which match the main settings app. B.1.

Visual analysis. The paper compares cosine-similarity matrices on 50 COCO samples; the stage-1 model gives a clearer diagonal than E5-V fig. 3 §4.3. Stage 2 shows a larger gap in training loss and gradient norm for hard negatives than for random or easy ones fig. 4 §5. Top-k token visualisations are in fig. 5 and fig. 6; negative examples are in fig. 7 app. C.2.

Paper vs code

The code is the official repository at github.com/deepglint/UniME (MIT licence, commit 61db13d7bc334ec8fb9fa3735e535d981a27b62b). The paper links it in its title block. The repository has loss.py (the stage-2 loss), evaluation scripts, and a quick-start example. It does not include the stage-1 distillation code, the training scripts or the stage-2 data pipeline.

• Temperature. The paper gives no value for τ §3.2 §3.3. The code defaults to 0.02 loss.py loss.py, and the top-k and threshold defaults are k=8 and margin 0.1 loss.py.
• Filtering and sampling. The code masks candidates above diagonal + threshold and the diagonal loss.py. It then takes top-k over the masked matrix loss.py. It does not duplicate negatives when too few remain, as §3.3 describes. See Pitfalls.
• Pool scope. The paper says hard negatives come from "each training batch" §3.3. The code all-gathers queries and keys across GPUs before ranking loss.py loss.py, so the pool is the global batch.
• Loss scale. The code multiplies the distributed loss by the world size loss.py. The paper does not state this.
• Similarity. The paper uses cosine similarity §3.3. The code uses a raw matrix product x @ y.T loss.py and assumes its inputs are already normalised. The quick start normalises both sides README.md.
• Prompts. The paper uses "Summary above sentences in one word:" §3.1. The README quick start uses "Summary above sentence in one word:" (singular) README.md. Its highlights section uses "Summarize the above sentences in one word." README.md.
• Stage 1 scope. The README says only the LLM is trained in stage 1 README.md, which matches §3.2.
• Metric. The code reports Recall@1, @5 and @10 for retrieval [code: self_evaluate/retrieval_self.py#L62-L66]. The paper's text says Precision §4.2, while its appendix and table captions say Recall@1 app. A.2 tab. 2.
• No discrepancies found in the stage-2 loss itself: the score is cross-entropy over the positive and k hard negatives at temperature τ, with the positive at index 0 loss.py loss.py, which matches eq. 4.

Pitfalls

• Stage 1 is not reproducible from the code. Only the stage-2 loss is released loss.py. Stage 1 must be implemented from eq. 2 and §3.2 alone. The KL direction and the teacher-embedding normalisation follow the text, and the authors do not state the batch or sampling details beyond the table values.
• Eq. 2 is ambiguous as printed. The numerator is e_{s_i}^\top e_{s_i}, a self-similarity that equals 1 for unit vectors, while the denominator sums e_{s_j}^\top e_{s_i} over j. The intended pairwise term is not stated. We read it as a row-wise softmax over the batch similarity matrix, as in the pseudocode above. Check this with the authors before relying on it.
• Fewer than k valid negatives. §3.3 says to duplicate existing hard negatives to keep k. The code masks filtered candidates and the diagonal with -\infty loss.py, then calls torch.topk loss.py. If fewer than k candidates remain unmasked, topk returns masked entries, which can include the positive. This is rare at batch 1024, but heavy filtering makes it likelier: β=-0.1 filters 81.7% of samples tab. 4.
• Temperature is a free choice. τ is not stated for either stage. Code uses 0.02 for stage 2 loss.py. The stage-1 value is not in the repo.
• Gradients through all-gather. The code gathers embeddings with dist.all_gather into fresh tensors loss.py. As far as we know, plain torch.distributed.all_gather is not differentiable, so gradients may not reach keys held on other ranks. Check this before relying on it for multi-GPU training. The paper says nothing about it.
• Trainable parameters in stage 2 are unclear. §3.4 says QLoRA is used "for parameter-efficient fine-tuning of all parameters within the MLLM". Stage 1 trains only the LLM §3.2. The vision encoder and projector are not clearly frozen or trained in stage 2. The LoRA rank is 16 tab. 6.
• Only Phi-3.5-V has ablations. The β, k, stage and LoRA-rank studies are on Phi-3.5-V tab. 3 tab. 4 tab. 5 tab. 8. Their settings may not carry over to LLaVA-1.6 or LLaVA-OneVision.
• LLaVA-OneVision settings are missing. The best MMEB row uses it tab. 1, but the training table covers only Phi-3.5-V and LLaVA-1.6 tab. 6. Its learning rate, batch and steps are not stated.
• Backbone count. §4.1 says "three" MLLMs, then names two (Phi-3.5-Vision and LLaVA-1.6). The tables add LLaVA-OneVision.
• Metric wording. The text says Precision for retrieval §4.2. Appendix A.2 says Recall@1 app. A.2. Table 2 is Recall@1 tab. 2. The code reports Recall@1, @5 and @10. Compare numbers only within one metric.
• Missing figure. The stage-1 text refers to a figure labelled fig:method_distillation §3.2, but no such figure appears in the source. That reference may render as unresolved.
• Notation clash. k is both the number of hard negatives and "the set of all hard negatives" §3.3.
• Headline gap does not match the table. §4.3 says UniME (LLaVA-1.6) beats VLM2Vec by 10.3%. tab. 1 gives 66.6 vs 63.3 overall, a 3.3-point gap. The 10.3 figure may refer to a different setting or metric. Quote the table.
• Baselines. Baseline MMEB numbers for CLIP, SigLIP, BLIP-2 and MagicLens are taken from VLM2Vec. E5-V, VLM2Vec and UniME were re-run by the authors, using only their best LLaVA-1.6 variants app. B.2. Compare with care.
• Costs the abstract hides. Stage 2 takes 26 h (Phi-3.5-V) and 37 h (LLaVA-1.6) on 8×A100 80GB §4.1 tab. 6. Stage 1 also needs offline teacher embeddings from NV-Embed V2 for every training sentence §3.2.
• Interleaved inputs. Embeddings of each part are summed element-wise §3.2. The repository has no code for this step. Treat it as unverified.

Lineage

• Builds on: E5-V (embedding prompt, unimodal contrastive training) §3.1 §3.2; VLM2Vec and the MMEB benchmark (task prompts, the MMEB evaluation protocol, GradCache) §3.1 §3.4 §4.2; NV-Embed V2 (teacher model) §3.2; QLoRA §3.4. None of these is compiled in this wiki yet.
• Compared against: CLIP, OpenCLIP, EVA-CLIP, SigLIP, BLIP-2, MagicLens, E5-V, VLM2Vec tab. 1 tab. 2.
• Superseded by: not in this wiki. The code README names UniME-V2 (arXiv:2510.13515) as a follow-up README.md.