---
kind: concept
slug: modality-imbalance
title: Modality imbalance
summary: >-
  When a multimodal network learns mostly from its stronger input and under-uses the weaker one. One method so far: prototype-based rebalancing (PMR). It adds a per-class prototype loss to the weaker modality and an early entropy term to the stronger one, with no change to the architecture or fusion. Author-reported gains on three datasets; one wiki card; released code omits the entropy term.
tags: [multimodal, training-dynamics, loss-function]
papers: ["2211.07089"]
related: [multimodal-fusion, prototype-learning]
status: draft
updated_at: 2026-11-06
---

## Overview

A network that takes several input streams, such as audio and video, is trained
with one loss over the fused output. The authors argue that the stronger stream
then dominates the gradient and the weaker stream's branch is left under-trained,
so the combined model does not use all of its inputs [arxiv:2211.07089 §1]
[arxiv:2211.07089 §3]. On CREMA-D, the gradient direction from the fused output
drifts away from each single-modality guidance direction during training
[arxiv:2211.07089 §3]. The shared invariant of the remedies is a per-modality
signal that tells the trainer which branch is lagging. The variants below differ
in how that signal is measured and applied.

## Variants

### Prototype-based rebalancing (PMR, full method)

Each modality gets class prototypes, which are mean feature vectors per class,
and a nearest-prototype classifier. A batch-level ratio of prototype scores
decides which modality is lagging [arxiv:2211.07089 §4.1] [eq. 8]. The lagging
modality receives an extra prototype cross-entropy loss. In the first epochs,
the leading modality receives an entropy regulariser that slows it down
[arxiv:2211.07089 §4.1] [arxiv:2211.07089 §4.2]. Card:
[PMR: Prototypical Modal Rebalance for Multimodal Learning](/papers/2211.07089).
Author-reported accuracy: CREMA-D concatenation 53.2 to 61.1 and CG-MNIST
summation 59.1 to 78.5 [arxiv:2211.07089 tab. 1].

### PMR without the entropy regulariser

The same method with the entropy term removed. The paper reports this as an
ablation: CREMA-D concatenation 61.1 with the regulariser, 60.3 without
[arxiv:2211.07089 tab. 2]. The gap is about one point from single runs, and the
paper states no variance, so the effect of the regulariser is not settled by
this card [arxiv:2211.07089 tab. 1] [arxiv:2211.07089 tab. 2].

### Released-code variant

The official repository implements the prototype loss and the imbalance ratio
but not the entropy regulariser. It also uses a hard switch between the two
modalities where the paper uses a clipped ramp. The card's Paper vs code
section lists these differences and the code locations
[arxiv:2211.07089 §4.1] [arxiv:2211.07089 eq. 11].
Use this variant only if you need to match the released results, and say so in
your write-up. Card: [PMR](/papers/2211.07089).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Two modalities, one visibly weaker, labelled classification data, any fusion | Full PMR | Reported gains with concatenation, film and summation fusion [arxiv:2211.07089 tab. 1]. The method needs only each encoder's output [arxiv:2211.07089 §4.1]. |
| Gated fusion | Expect little or no gain | Reported change on AVE is 63.5 to 62.7 [arxiv:2211.07089 tab. 1]. |
| Intermediate fusion (MMTM) or CentralNet | Try it, expect smaller gains | Reported gains are 51.6 to 55.1 and 50.2 to 52.9 on CREMA-D [arxiv:2211.07089 tab. 3]. The card did not trace these runs into the code. |
| Little labelled data for prototypes | Keep the prototype subset near 10% | On AVE, 1% of the data gives 62.1 against a 65.4 baseline; 10% gives 67.1 [arxiv:2211.07089 tab. 5] [arxiv:2211.07089 tab. 1]. |
| Adaptive optimiser (Adam) | Use with care | The derivation assumes one learning rate for all parameters. Reported gain with Adam is 59.8 to 65.3, which is empirical only [arxiv:2211.07089 §5.4] [arxiv:2211.07089 tab. 6]. |
| Tight compute budget | Check the cost first | The prototypes need a forward pass over the subset every epoch, which the paper names as the main extra cost [arxiv:2211.07089 §5.4]. |
| Need to match released results | Released-code variant | The code lacks the entropy term [arxiv:2211.07089 §4.2]. |
| Need a method that changes the fusion | Other families | The paper compares against Modality-Drop, Gradient-Blending and OGM-GE [arxiv:2211.07089 tab. 2]. Those papers have no card here, so this page does not describe them. |

## Implementation sketch

The common core is a per-modality prototype classifier, an imbalance ratio, and
a loss that adds a term for the lagging modality. The released code and the
paper differ in the coefficients and the entropy term; the sketch follows the
paper's printed form.

```pseudocode
# Refresh per-modality class prototypes once per epoch  [arxiv:2211.07089 §4.1]
# Prototypes come from a subset of training data  [arxiv:2211.07089 §5.2]
protos0, protos1 = momentum_update(protos0, protos1, subset)

for x0, x1, y in batches:
    # Each encoder output, then the usual fused head  [arxiv:2211.07089 eq. 1]
    z0, z1 = enc0(x0), enc1(x1)
    logits = head(concat(z0, z1))

    # Per-modality prototype probabilities  [arxiv:2211.07089 eq. 7]
    p0 = softmax(-dist(z0, protos0))
    p1 = softmax(-dist(z1, protos1))

    # Imbalance ratio from batch sums of true-class scores  [arxiv:2211.07089 eq. 8]
    rho = sum(p0[y]) / sum(p1[y])

    # Prototype cross-entropy per modality  [arxiv:2211.07089 eq. 9]
    pce0 = cross_entropy(-dist(z0, protos0), y)
    pce1 = cross_entropy(-dist(z1, protos1), y)

    # Weights from rho, clipped ramp as printed  [arxiv:2211.07089 eq. 11]
    beta, gamma = modulation(rho)

    # Acceleration loss on the lagging modality  [arxiv:2211.07089 eq. 10]
    loss = cross_entropy(logits, y) + alpha * (beta * pce0 + gamma * pce1)

    # Early entropy term on the leading modality  [arxiv:2211.07089 eq. 13]
    if epoch < E_r:
        loss = loss - mu * (gamma * entropy(p0) + beta * entropy(p1))
    step(loss)
```

Guard the division in the ratio against a near-zero denominator. The released
code has no epsilon there, and a small denominator makes 1/rho large before
the clip [arxiv:2211.07089 §4.1] (see the card's Pitfalls).

## Contradictions

Only one wiki card covers this concept, so no two cards disagree. The paper and
its released code do disagree on several points, and both sides are cited here.

- **Entropy term.** The paper includes it [arxiv:2211.07089 eq. 13]. The
  released code has no entropy term [arxiv:2211.07089 §4.2].
- **Modulation shape.** The paper uses a clipped ramp [arxiv:2211.07089 eq. 11].
  The code uses a hard switch [arxiv:2211.07089 §4.1].
- **Modulation window.** The paper's algorithm applies the modulation for all
  training epochs [arxiv:2211.07089 alg. 1]. The code applies it for epochs 0 to
  100 by default, as the card's Paper vs code section records.
- **Learning-rate schedule.** The paper says the rate "gradually decays" from
  1e-3 to 1e-4 [arxiv:2211.07089 §5.2]. The code uses a single step down at
  epoch 70, per the card.
- **Prototype data.** The paper uses a subset of training data [arxiv:2211.07089
  §5.2]. The code uses the validation split for AVE and CG-MNIST, per the card.
- **Momentum.** The paper does not state the momentum coefficient eps
  [arxiv:2211.07089 eq. 12]. The released repository's default and its CREMA-D
  example script disagree on it, per the card.
- **Entropy term index.** The printed second entropy term in eq. 13 looks like a
  typo, so its exact form is unclear [arxiv:2211.07089 eq. 13]. Confirm before
  implementing it.

## Papers

- [PMR: Prototypical Modal Rebalance for Multimodal Learning](/papers/2211.07089): prototype-based detection of the lagging modality, a prototype loss on it, and an early entropy term on the leading one.