Modality imbalance

Overview

A network that takes several input streams, such as audio and video, is trained with one loss over the fused output. The authors argue that the stronger stream then dominates the gradient and the weaker stream's branch is left under-trained, so the combined model does not use all of its inputs 2211.07089 §1 2211.07089 §3. On CREMA-D, the gradient direction from the fused output drifts away from each single-modality guidance direction during training 2211.07089 §3. The shared invariant of the remedies is a per-modality signal that tells the trainer which branch is lagging. The variants below differ in how that signal is measured and applied.

Variants

Prototype-based rebalancing (PMR, full method)

Each modality gets class prototypes, which are mean feature vectors per class, and a nearest-prototype classifier. A batch-level ratio of prototype scores decides which modality is lagging 2211.07089 §4.1 eq. 8. The lagging modality receives an extra prototype cross-entropy loss. In the first epochs, the leading modality receives an entropy regulariser that slows it down 2211.07089 §4.1 2211.07089 §4.2. Card: PMR: Prototypical Modal Rebalance for Multimodal Learning. Author-reported accuracy: CREMA-D concatenation 53.2 to 61.1 and CG-MNIST summation 59.1 to 78.5 2211.07089 tab. 1.

PMR without the entropy regulariser

The same method with the entropy term removed. The paper reports this as an ablation: CREMA-D concatenation 61.1 with the regulariser, 60.3 without 2211.07089 tab. 2. The gap is about one point from single runs, and the paper states no variance, so the effect of the regulariser is not settled by this card 2211.07089 tab. 1 2211.07089 tab. 2.

Released-code variant

The official repository implements the prototype loss and the imbalance ratio but not the entropy regulariser. It also uses a hard switch between the two modalities where the paper uses a clipped ramp. The card's Paper vs code section lists these differences and the code locations 2211.07089 §4.1 2211.07089 eq. 11. Use this variant only if you need to match the released results, and say so in your write-up. Card: PMR.

When to use which

Implementation sketch

The common core is a per-modality prototype classifier, an imbalance ratio, and a loss that adds a term for the lagging modality. The released code and the paper differ in the coefficients and the entropy term; the sketch follows the paper's printed form.

# Refresh per-modality class prototypes once per epoch  [arxiv:2211.07089 §4.1]
# Prototypes come from a subset of training data  [arxiv:2211.07089 §5.2]
protos0, protos1 = momentum_update(protos0, protos1, subset)

for x0, x1, y in batches:
    # Each encoder output, then the usual fused head  [arxiv:2211.07089 eq. 1]
    z0, z1 = enc0(x0), enc1(x1)
    logits = head(concat(z0, z1))

    # Per-modality prototype probabilities  [arxiv:2211.07089 eq. 7]
    p0 = softmax(-dist(z0, protos0))
    p1 = softmax(-dist(z1, protos1))

    # Imbalance ratio from batch sums of true-class scores  [arxiv:2211.07089 eq. 8]
    rho = sum(p0[y]) / sum(p1[y])

    # Prototype cross-entropy per modality  [arxiv:2211.07089 eq. 9]
    pce0 = cross_entropy(-dist(z0, protos0), y)
    pce1 = cross_entropy(-dist(z1, protos1), y)

    # Weights from rho, clipped ramp as printed  [arxiv:2211.07089 eq. 11]
    beta, gamma = modulation(rho)

    # Acceleration loss on the lagging modality  [arxiv:2211.07089 eq. 10]
    loss = cross_entropy(logits, y) + alpha * (beta * pce0 + gamma * pce1)

    # Early entropy term on the leading modality  [arxiv:2211.07089 eq. 13]
    if epoch < E_r:
        loss = loss - mu * (gamma * entropy(p0) + beta * entropy(p1))
    step(loss)

Guard the division in the ratio against a near-zero denominator. The released code has no epsilon there, and a small denominator makes 1/rho large before the clip 2211.07089 §4.1 (see the card's Pitfalls).

Contradictions

Only one wiki card covers this concept, so no two cards disagree. The paper and its released code do disagree on several points, and both sides are cited here.

• Entropy term. The paper includes it 2211.07089 eq. 13. The released code has no entropy term 2211.07089 §4.2.
• Modulation shape. The paper uses a clipped ramp 2211.07089 eq. 11. The code uses a hard switch 2211.07089 §4.1.
• Modulation window. The paper's algorithm applies the modulation for all training epochs 2211.07089 alg. 1. The code applies it for epochs 0 to 100 by default, as the card's Paper vs code section records.
• Learning-rate schedule. The paper says the rate "gradually decays" from 1e-3 to 1e-4 2211.07089 §5.2. The code uses a single step down at epoch 70, per the card.
• Prototype data. The paper uses a subset of training data [arxiv:2211.07089 §5.2]. The code uses the validation split for AVE and CG-MNIST, per the card.
• Momentum. The paper does not state the momentum coefficient eps 2211.07089 eq. 12. The released repository's default and its CREMA-D example script disagree on it, per the card.
• Entropy term index. The printed second entropy term in eq. 13 looks like a typo, so its exact form is unclear 2211.07089 eq. 13. Confirm before implementing it.

Papers

• PMR: Prototypical Modal Rebalance for Multimodal Learning: prototype-based detection of the lagging modality, a prototype loss on it, and an early entropy term on the leading one.