Multimodal fusion

Overview

Multimodal fusion combines inputs of different kinds into one output. Twenty-one cards on this page describe fusion designs, and four more complete it. They cover utterance-level sentiment and humour from words, voice and face 2005.03545 §3.1; sentiment regression with mutual-information terms 2109.00412 §3.2; 3D dance generation from music and a short motion seed 2101.08779 §4.1; contact-rich robot manipulation from a camera and a fingertip tactile sensor 2507.15062 §4.2; visual grounding with one joint transformer 2104.08541 §3.2; image-text question answering with two transformer streams and no detector 2111.02387 §3.1; question answering over a 3D indoor scan, fusing words with object proposals 2112.10482 §4; RGB-D salient object detection with a depth gate 1907.06781 §4.2; multi-view classification with an uncertainty value 2204.11423 §3.3; emotion detection from face and voice with a disagreement check 2605.30099 §III.E; and one network that fuses cameras, event cameras and LiDAR for several driving tasks 2606.02979 §3.1; a microphone on a robot gripper, fused with the camera for contact-rich manipulation 2406.19464 §3.2; and an audio prompt mapped into the goal space of a Minecraft agent 2412.00949 §3.1. Three more designs come from other sensor and text settings: radar points added to a camera detector 2011.04841 §4.3, text-based outer products of audio and video with a correlation loss 1911.05544 §3, and a document encoder whose attention joins words, page pixels and layout in every layer 2106.11539 §3.1. The benchmark is Winoground 2204.03162 §3.1. It is an evaluation, not a fusion method. Three more cards add to the list. Hierarchical pairwise fusion of text, audio and video for sentiment 1806.06228 §3.4. One transformer over camera and LiDAR tokens for driving 2207.14024 §3.2. And a survey that sorts LiDAR-camera fusion in 3D detection into early, intermediate and late stages 2206.09474 §5.1. Three more designs follow: a shared code for any subset of views 2011.06170 §2.1, two Swin streams with an edge branch for salient object detection 2204.05585 §3.2, and one transformer over image, text and lab tokens for clinical diagnosis 2306.00864 p. 10. The four cards that complete the page are the survey and the benchmark above, a movie dataset whose story-retrieval baseline is a design in its own right 2007.10937 §4.5, and a position paper on sentiment analysis that reports no experiments 2005.00357 §1. That paper calls MOSI, MOSEI and MELD too small, with weak annotator agreement 2005.00357 §3.2.2. For other settings, such as a second sensor added to an image, see image fusion and multimodal language models.

The difficulty is that the inputs have different distributions. The authors of the sentiment paper call the gap between them a modality gap. They argue that fusion has to bridge this gap, and that most earlier methods leave it to the fusion step itself 2005.03545 §1.

The designs on this page differ in what they fuse and where. Some split each modality into a shared and a private code 2005.03545 §3.3. Some concatenate token sequences into one transformer 2101.08779 §4.1 or add a learned read-out token 2104.08541 §3.2. InterFuser joins camera and LiDAR tokens in one encoder 2207.14024 §3.2. Some let two embeddings attend to each other 2507.15062 §4.2. Two designs run a stream per input and let the streams read each other layer by layer: the image-text model in 2111.02387 §3.1 and the 3D question model in 2112.10482 §4. Others keep one decision per input and combine the decisions: a depth gate at test time 1907.06781 §4.2, an evidential rule that lets uncertain views count less 2204.11423 §3.3, or a rank check between two classifiers 2605.30099 §III.E. Two designs keep one fused representation but shape it through training: a mutual-information term that rewards the fused vector for keeping information from each input 2109.00412 §3.2, and a shared bottleneck that all of several sensor streams and tasks pass through 2606.02979 §3.1.

CenterFusion takes a different route: radar is not a token stream at all. Its matched radar depth and velocity become image-shaped channels for a second stage of the detector 2011.04841 §4.3. The document model works inside each layer. It keeps a text branch and a visual branch, and the two spatial terms share their query and key matrices 2106.11539 §3.1. ICCN forms an outer product of the text vector with each audio or video vector before any network sees it 1911.05544 §3.

Two designs add audio to a visual pipeline. ManiWAV puts one vision token per frame and one audio token into a single transformer layer 2406.19464 §3.2. STEVE-Audio does not fuse features at all. It aligns audio and video in one contrastive space, then maps audio into the goal space that an existing policy already uses 2412.00949 §3.1 §3.4.

Variants

Shared and private codes with attention fusion

Each modality is encoded into one vector. That vector gives a shared code from an encoder used for all modalities, and a private code from an encoder per modality 2005.03545 §3.3. A one-layer self-attention block fuses the six codes, and a head makes the prediction 2005.03545 §3.4. Training adds three regularisers to the task loss: a distribution-matching term on the shared codes, a term that keeps shared and private codes apart, and a reconstruction term 2005.03545 §3.5. The full model reports MOSI MAE 0.783 and correlation 0.761, MOSEI MAE 0.555, and UR-FUNNY accuracy 70.6 2005.03545 tab. 4. Card: MISA.

The next four variants are ablations of this design, all from the same table.

Shared codes only for fusion

The private codes are still trained but are not fused. Reported: MOSI MAE 0.850, MOSEI MAE 0.555, UR-FUNNY 69.8 2005.03545 tab. 4. The paper's label for this row is MISA-iFusion. It matches the full model on MOSEI MAE and is worse on MOSI.

Private codes only for fusion

The shared codes are still trained but are not fused. Reported: MOSI MAE 0.858, MOSEI MAE 0.563, UR-FUNNY 70.1 2005.03545 tab. 4. Label MISA-sFusion. Of the two single-type fusion variants, this one is the weaker on MOSI.

Invariant codes only

The paper's label is MISA-inv. Reported: MOSI MAE 0.811, MOSEI MAE 0.561, UR-FUNNY 68.8 2005.03545 tab. 4. It is the lowest UR-FUNNY score of the subspace variants on the page.

No subspace split

Each modality has its own encoder and no shared or private code is built. Label MISA-base. Reported: MOSI MAE 0.810, MOSEI MAE 0.568, UR-FUNNY 69.2 2005.03545 tab. 4.

Early fusion by token concatenation

Motion and music are each embedded and passed through their own two-layer transformer, with width 800 and 10 heads. The two token sequences are then concatenated and sent through one 12-layer cross-modal transformer with full attention and no mask 2101.08779 §4.1 2101.08779 §5.2.1. Music is a 4-second window of 35 audio features per frame, and the motion window is 120 frames 2101.08779 §5.2.1. Card: FACT.

The authors compare three layouts, all trained with the same future-20 objective 2101.08779 tab. 4:

• No fusion, a 14-layer motion transformer only: FID_k 45.66, FID_m 13.27, BeatAlign 0.228. The paper computes BeatAlign with the music paired to the input motion, as its table caption notes.
• Late fusion, labelled 13 + 1 layers: FID_k 45.76, FID_m 14.30, BeatAlign 0.234.
• Early fusion, the FACT design: FID_k 35.35, FID_m 12.39, BeatAlign 0.241.

Late fusion is worse than no fusion on FID_k and FID_m and better on BeatAlign. The early layout is best on all three. The numbers are the authors', from one dataset, and the card does not report seed counts for the ablation. The released code prints an FID_k near 38 rather than 35.35, so a re-run may not match 2101.08779 §5.2.2. See the card's Paper vs code section.

Cross-attention between image and touch

A camera image goes through a CLIP ViT-B/16 that is fine-tuned, and its [CLS] token is the visual embedding. Two fingertip tactile images are stacked into one image and go through a three-layer CNN to a 768-dimension vector. Each embedding then attends to the other with multi-head cross-attention and LayerNorm. The fused representation is the concatenation of the two updated embeddings, which is 2d in size 2507.15062 §4.1 2507.15062 §4.2 2507.15062 eq. 2 2507.15062 eq. 3 2507.15062 eq. 4. The paper's text says "two rounds" of cross-attention. The equations show one step in each direction, and the card records that the released code matches the equations 2507.15062 §4.2. Card: Touch in the Wild.

The authors use a masked tactile reconstruction objective instead of contrastive alignment. Their reason is that contrastive alignment tends to suppress the fine geometry of the tactile signal 2507.15062 §4.2.

The paper's real-robot results, 20 trials per cell 2507.15062 tab. 1. Test-tube whole task, the cleanest comparison:

• Vision only: 0.25.
• Concatenation instead of cross-attention: 0.50.
• Cross-attention without visuo-tactile pretraining: 0.70.
• Cross-attention with pretraining: 0.85.

The same ordering holds on the fluid-transfer and pencil whole tasks. On whiteboard erasing, concatenation scores below vision only (0.45 against 0.55), and pretraining is 0.70 against 0.60 without it [arxiv:2507.15062 tab. 1]. Each cell is 20 trials, so one trial is 0.05, and the paper gives no error bars.

Joint transformer with a read-out token

TransVG puts the visual tokens, the language tokens and one learned [REG] token into a single stack of six post-norm encoder layers. The box is read from the output at the [REG] position 2104.08541 §3.2 [arxiv:2104.08541 eq. 4]. The visual tokens come from a ResNet branch initialised from DETR, and the language tokens from the first 12 layers of BERT-base. Both are projected to 256 dimensions 2104.08541 §3.2 2104.08541 §4.2. The output is a single box, so no proposals or anchors are needed 2104.08541 §3.2. Card: TransVG.

The authors fit the box with a smooth-L1 term plus a GIoU term 2104.08541 §3.3 2104.08541 eq. 5. The card's Paper vs code section records that the released code uses plain L1 for the coordinate term.

Author-reported top-1 accuracy at IoU 0.5, with ResNet-101 [arxiv:2104.08541 tab. 1] tab. 2: 70.73 on ReferItGame test, 79.10 on Flickr30K test and 78.35 on RefCOCO testB. The authors' reproduction with the released code gives 76.94 on RefCOCO testB 2104.08541 tab. 2. Their ablation on ReferItGame shows the effect of the branch transformers. Accuracy is 64.24 with neither, 66.78 with only the linguistic branch transformer, 68.48 with only the visual one, and 69.76 with both 2104.08541 tab. 4. These numbers use ResNet-50.

Dual-stream co-attention with pre-trained encoders

METER pairs a pre-trained text encoder with a pre-trained vision transformer. The default is RoBERTa for text and CLIP ViT-224/32 for images, and the best reported models use CLIP ViT-224/16 or Swin 2111.02387 §3.1 2111.02387 §4.1.1. Each stream has six layers. A layer holds self-attention, cross-attention over the other stream, and a feed-forward block. The fused output is the two pooled first tokens, one per stream 2111.02387 §3.3. The model is pre-trained on about 4M images from four caption datasets, for 100k steps at batch size 4096 2111.02387 §3.3 2111.02387 app. A. Card: METER.

The paper compares this with a single merged transformer over both token sequences. Merged attention scores lower on VQAv2 test-dev, 74.00 against 74.98, and on Flickr30k zero-shot image retrieval, 57.46 against 66.08 2111.02387 tab. 6. The encoder-decoder version is weaker than the encoder-only co-attention model on all three numbers in that table.

Author-reported VQAv2 test-std accuracy is 77.64 for the CLIP model with about 4M images. The authors list VinVL LARGE at 76.60; VinVL uses region features from a detector 2111.02387 tab. 8. A 14M-image version with a larger backbone reaches 80.54 2111.02387 tab. 10. The paper does not state the compute cost of that run. The inference time is about 53 ms per VQA instance on one V100 2111.02387 app. B 2111.02387 tab. 12. The ViLT column in that table is copied from the ViLT paper and was measured on other hardware, so compare the rows within the METER columns only.

The ablations give the authors' reasons for the design. Pre-training with masked language modelling and image-text matching gives 74.98 on VQAv2 test-dev. Adding masked patch prediction, in either of two forms, lowers it to 74.01 or 74.21 2111.02387 tab. 7. Raising the fine-tuning image size from 224 to 576 adds about 3 points to VQA for the 224/32 model and about 1 point for the 224/16 model 2111.02387 §4.1.2. The ablations use test-dev and zero-shot Flickr30k retrieval, not test-std 2111.02387 §4.1.

Object proposals and words in one transformer

ScanQA answers a free-form question about a 3D indoor scan and points to the objects the question is about 2112.10482 §3.1. The scan is encoded as point features, and VoteNet with a PointNet++ backbone proposes 256 objects 2112.10482 §4. Question words are embedded with GloVe and passed through a one-layer LSTM. The paper calls this LSTM bidirectional 2112.10482 §4. Two transformer encoder layers process the words and two decoder layers let the object proposals attend to them, following the MCAN blocks the authors cite 2112.10482 §4. An attention step then merges both streams into one vector.

Three heads read this design. A two-layer head scores each proposal against the question. A second head predicts one of 18 ScanNet object classes. A third predicts the answer with a binary cross-entropy loss, so several answers can be correct. The loss is the plain sum of the four terms, and the paper sets each term's weight to one 2112.10482 §4. Card: ScanQA.

Author-reported EM@1 on the test split with objects is 23.45, against 22.31 for the best 2D baseline, RandomImage with MCAN 2112.10482 tab. 3. On the split without object annotations it is 20.90, against 20.82 for the same baseline 2112.10482 tab. 3. The paper's oracle that uses the true object reaches 25.34 on EM@1 2112.10482 tab. 3. Removing parts of the model lowers the score on the split with objects. With only the answer head, EM@1 is 12.16. Adding the object-class head gives 18.31, adding the localization head gives 20.46, and all three give 23.45 2112.10482 tab. 4.

The split without objects has no localization labels, so the box metrics do not apply to it 2112.10482 §3.3. The paper's best human worker scores 51.6 EM@1 on a sample of test questions 2112.10482 §5.2. The authors describe that as one worker on a sample, so use it as a rough ceiling only. The paper also does not state its training hardware or time 2112.10482 §5. The dataset is built from ScanNet scenes and ScanRefer captions 2112.10482 §3.2. The terms of use of those sources were not checked for this page, so check them before reusing the data.

Depth gate at test time

Three streams are trained separately: one on colour, one on colour plus depth, and one on depth alone 1907.06781 §4.1 1907.06781 §4.3. At test time, the colour-plus-depth map and the depth-only map are compared by their mean absolute difference. If the difference is at most t = 0.15, the colour-plus-depth map is kept. Otherwise the colour-only map is used 1907.06781 §4.2 1907.06781 eq. 2 1907.06781 eq. 3. The paper chose t after trying several values, on data it does not name 1907.06781 §4.3. Card: D3Net.

The paper's benchmark has seven test sets and 18 models [arxiv:1907.06781 §5.1]. D3Net has the highest S-measure on six of the seven sets. CPFP is ahead on LFSD 1907.06781 tab. 4. On NJU2K the two S-measures are 0.900 and 0.879 1907.06781 tab. 4. Against the colour-plus-depth stream alone, the gate changes S-measure by 0.001 to 0.002 on NJU2K, NLPR, STERE and DES, and leaves SSD unchanged. It is lower on LFSD (0.825 against 0.836) and on SIP (0.860 against 0.862) 1907.06781 tab. 5. The card's text names only the four datasets where the gate helps. The card records no spread across runs, so these gaps are small single-run differences.

Evidential combination with a reduced Dempster rule

Each view's network outputs non-negative evidence for each class. Evidence plus one gives Dirichlet parameters. From these each view gets belief masses per class and an uncertainty mass of K over the total strength S, where K is the number of classes 2204.11423 §3.2 2204.11423 eq. 11. Views are combined pairwise with a reduced Dempster rule. A view that is uncertain contributes less to the result, and the joint uncertainty is the product of the two uncertainties divided by one minus the conflict between them 2204.11423 §3.3 2204.11423 eq. 13. Training uses one variational term per view and one for the fused opinion, each a fit term plus a KL penalty towards the uniform Dirichlet. The KL weight is ramped up over training 2204.11423 §3.1 2204.11423 §3.6 [arxiv:2204.11423 alg. 1]. ETMC adds a pseudo-view built from the concatenated view features, and fuses it in the same way 2204.11423 §3.5. Card: TMC and ETMC.

Author-reported results 2204.11423 tab. 1 tab. 3 tab. 4 tab. 5:

• On PIE, accuracy is 91.99 for TMC and 93.75 for ETMC. The best single-view baseline, EDL, reaches 84.36.
• On SUN RGB-D fusion, ETMC reaches 61.3 and TMC 60.7. The best earlier method in the paper's table is at 59.5.
• On NYUD2 fusion, ETMC reaches 72.5. The best earlier method is at 70.9, and TMC at 70.0.
• Compute is 3.64 GFLOPs for both TMC and ETMC, the same as late fusion.

The card records that the released RGB-D scripts pick the checkpoint and the learning-rate schedule on the test loader, so RGB-D numbers may be optimistic.

Decision-level agreement check

Two classifiers, one on video frames and one on audio features, each give scores over seven emotions. The method is applied on top of their outputs 2605.30099 §III. Each short sub-video gets a top-three emotion list from each modality. The method returns the first emotion that appears in both lists. If no emotion matches, it flags sarcasm 2605.30099 §III.E 2605.30099 alg. 1. Card: Emotion and sarcasm.

The author-reported accuracy for the full model is 95.7 [arxiv:2605.30099 tab. 3]. The paper's own tables disagree on the CNN-only and post-check per-class numbers, and the sarcasm claim rests on one public video 2605.30099 §III.F 2605.30099 fig. 10. Use this design as a pattern. The card does not show a clean measured gain from the check alone.

Mutual-information terms added to the task loss

MMIM keeps the usual regression loss and adds two mutual-information terms. The text is the anchor. An input-level term bounds the information it shares with the visual input and with the acoustic input, using a Barber-Agakov bound with a learned predictor and a mixture-based entropy estimate [arxiv:2109.00412 §3.4] eq. 8. A fusion-level term asks the fused vector to predict each of the three modality vectors, with a contrastive loss over the batch [arxiv:2109.00412 §3.5] eq. 11. Training alternates two stages: the predictors are fitted first, then the rest of the network is trained on the combined loss 2109.00412 §3.6 alg. 1. Card: MMIM.

Author-reported results on CMU-MOSEI 2109.00412 tab. 2: MAE 0.526, correlation 0.772 and Acc-7 54.24. The same table lists MISA at 0.568 and Self-MM at 0.530 on MOSEI MAE, both taken from other papers. On CMU-MOSI the paper reports MAE 0.700 and correlation 0.800. In the MOSEI ablation, removing all three input-level terms gives MAE 0.541, and removing the fusion-level term gives 0.543. The full model gives 0.526 2109.00412 tab. 4. These are single runs with no spread shown.

The released code does not match the paper's defaults in several places. It trains the MI terms only when a flag is set, and it chooses the reported epoch on the test set. The card's Paper vs code section lists these, and it should be read before reimplementing.

Shared bottleneck for multi-sensor driving perception

One network takes four RGB views, four event-camera views and a LiDAR stack of 15 height channels. Each input has its own encoder. The first bottleneck fuses the RGB, event and LiDAR features and feeds the depth, segmentation and LiDAR-segmentation decoders. A second bottleneck takes their outputs and feeds the bird's-eye-view decoder 2606.02979 §3.1 fig. 2. The loss weights are adapted with a modified GradNorm, updated once per epoch 2606.02979 §3.3 alg. 1. Card: Compact perception.

Author-reported total metric, lower is better, for the 15-channel model with the adaptive weights: 1.393 on set A, 1.086 on set B and 1.069 on nuScenes-lidarseg. The combined single-task baselines score 1.430, 1.160 and 1.124 on those sets. On set C the baselines are better, 0.976 against 0.999 [arxiv:2606.02979 tab. 2]. The simulated-set models have about 2.5 million parameters, against about 137 million for the combined baselines 2606.02979 tab. 3. The table has no variant with the fusion step removed, so these numbers compare whole models and do not isolate the fusion. The authors credit the depth and segmentation gains to the first bottleneck 2606.02979 §5.1. The baselines are retrained by the authors, and the ± values are variances, not standard errors 2606.02979 tab. 2.

Vision and audio tokens in one transformer layer

ManiWAV encodes two camera frames with a CLIP ViT-B/16 that is trained end to end. Its audio is a log-mel spectrogram encoded by an audio spectrogram transformer (AST) trained from scratch 2406.19464 §3.2. One transformer layer with 8 heads takes the vision feature of each frame and the audio feature. Its output is projected to 768 dimensions, and the poses of the last two timesteps are appended 2406.19464 §3.2 app. A.2. A diffusion head conditioned on this vector predicts a 16-step action chunk 2406.19464 §3.2. Card: ManiWAV.

Two training choices come from the paper. Background clips and robot-motor noise are each overlaid on the audio with probability 0.5, so the policy learns to ignore sounds that do not carry the task 2406.19464 §3.2. The paper also reports that a transformer encoder beats a CNN audio encoder, pretrained or not 2406.19464 §3.2.

Author-reported success on the robot, with 20 rollouts per row, is 85% for wiping against 40% for vision only and 35% for an MLP fusion head 2406.19464 §4.2 app. B.2.3. On flipping it is 90% against 25% for vision only and 60% for an MLP policy 2406.19464 §4.1 app. B.2.4. The paper's pouring rates cannot come from 12 trials, so this page does not use them 2406.19464 app. B.2.1.

Audio aligned to a frozen goal space

STEVE-Audio adds audio prompts to the Minecraft agent STEVE-1 without changing the agent. Frozen MineCLIP video features and frozen AST logits each go through a trainable mapping network into one 512-dimension space. Only these two networks are trained, with a CLIP-style contrastive loss 2412.00949 §3.1 §3.3. A conditional VAE then maps the audio embedding to the visual goal embedding that the policy already uses 2412.00949 §3.4. Card: STEVE-Audio.

Author-reported, on six item-collection tasks with 10 seeds each: audio beats the visual prompt on four tasks and the text prompt on five. It is weaker on the three "place" tasks 2412.00949 §4.1 §4.2.1. The results are shown only as ratios in a figure, not as counts 2412.00949 §4.1 fig. 4. The paper's conclusion says "outperforms", while its results text reports some lower ratios, so use the results text 2412.00949 §5.

Radar points as image-shaped channels

CenterFusion adds radar to a camera-only detector built on centre points 2011.04841 §4.1. Each radar point is first expanded into a fixed-size pillar §4.2. For each detected object, the preliminary box defines a frustum, and inside it a 3D region of interest built from the estimated depth, size and rotation. The radar point closest to the object is kept §4.2. Its depth and two velocity components are written as three channels over the object's image region and concatenated to the image features. Second-stage heads then recompute depth and rotation and predict velocity and attributes §4.3 fig. 1. Card: CenterFusion.

Author-reported on the nuScenes validation split with a DLA backbone: NDS 0.453 against 0.328 for the camera-only baseline, and mAP 0.332 against 0.306 2011.04841 tab. 1. The velocity error is the largest change, mAVE 0.540 against 1.426 tab. 1. In the ablation, pillar expansion alone raises NDS by 15.4% and frustum association alone by 25.9%, both relative to the baseline tab. 3. On the test split the paper reports NDS 0.449 against 0.400 for CenterNet with an Hourglass backbone, but mAP is lower, 0.326 against 0.338 tab. 1. The paper gives the test-time enlargement of the region as 20% in the radial direction, while the released code sets that factor to zero 2011.04841 §5 opts.py. Use the card's Paper vs code section before reproducing the numbers.

Layer-wise multimodal self-attention with shared spatial terms

DocFormer keeps a visual branch and a text branch in every encoder layer. The visual branch works on a grid of ResNet50 tokens and the text branch on word-piece tokens 2106.11539 §3.1. The attention score of each branch is the sum of a content term, two relative-offset terms and a spatial term. The spatial query and key matrices are shared by the two branches, so the branches learn cross-modal spatial interactions §3.1 eq. 6 eq. 7. The two branch outputs are summed, and the visual and spatial features are passed to every layer, not only the first §3.1 §3. Card: DocFormer.

The model is pre-trained on 5M IIT-CDIP pages with masked language modelling, an image-reconstruction loss and a loss that asks whether the text describes the image, then fine-tuned end to end 2106.11539 §3.2. Author-reported: FUNSD F1 83.34 for the base model with image, text and spatial inputs, against 81.21 for BROS-base with text and spatial inputs only 2106.11539 tab. 1. On RVL-CDIP, 96.17 for DocFormer-base against 95.25 for LayoutLMv2-base tab. 2. The ablations use 1M pre-training pages for 5 epochs, so they are not comparable with these headline numbers §4.5. In that ablation, adding the multi-modal attention to a spatial text model raises FUNSD F1 from 73.01 to 76.90 tab. 9. Sharing the spatial weights between the branches scores 76.9 on FUNSD against 75.58 for separate weights, with fewer parameters tab. 5. No official code was found, so an implementation has to be written from the card's pseudocode and equations.

Outer products with a canonical-correlation loss

ICCN takes an outer product of the BERT text vector with the final state of an LSTM for each of audio and video, so every text dimension meets every audio or video dimension 1911.05544 §3 eq. 5. Audio and video each pass through a 1D convolution and the LSTM before this step §3. Each interaction matrix goes through a 2D CNN, and a deep CCA loss trains the two CNNs so that their outputs are maximally correlated §3 eq. 4 alg. 1. After training, the two CNN outputs are concatenated with the text vector and passed to a small downstream MLP §3 §4. Card: ICCN.

Author-reported on CMU-MOSI, ICCN is best on all five metrics, with binary accuracy 83.07 and MAE 0.862 1911.05544 tab. 1. On CMU-MOSEI it is best on MAE and seven-class accuracy, and MFM is higher on binary accuracy (84.40 against 84.18) tab. 1. Removing the text vector and training DCCA on raw audio and video lowers MOSI binary accuracy to 82.13 tab. 3. The paper's features are BERT text, COVAREP audio and FACET video §4. The released MOSEI export selects the reported run on the test set, and its best printed run gives MAE 0.5693 rather than the table's 0.565. The card's Paper vs code section lists these and the paper's two readings of its CCA loss.

Winoground 2204.03162 §3.1 tests whether an image-text model separates two captions made of the same words in a different order, each paired with one of two images. It is a probe of compositional grounding, not a model. The paper defines three binary scores per example: text, image and group, where the group score needs all four pairings to be right 2204.03162 §3.2. Four expert annotators wrote the 400 examples, and an example was kept only if annotators preferred the correct pairings to the swapped ones [arxiv:2204.03162 §3.1]. The three scores apply to any image-text scorer, not only to the fusion designs on this page.

Author-reported results on 400 examples 2204.03162 tab. 3:

• Human group score 85.50; chance 16.67.
• Best model group score 14.50 (VinVL), and the other tested models are all below chance on the group score 2204.03162 §5.1.
• Text scores are above chance for several models, the best at 38.00 for UNITER large, against 89.50 for humans.

Use this as a diagnostic when you build or choose a fusion model that must bind attributes or relations to the right object. The compiled text does not state the score function that each model uses, so results depend on that choice. See the card's Pitfalls.

Hierarchical pairwise fusion with context

Text, audio and video features are first encoded per utterance. Each modality then runs through a GRU over the utterances of a clip, which gives context 1806.06228 §3.3. The three context vectors are mapped to a common width. Each pair of modalities is then fused dimension by dimension, with learned weights for each modality and a tanh 1806.06228 §3.4 eq. 1 eq. 2 eq. 3. A GRU runs over each pair sequence. The three pair outputs are combined the same way and passed through one more GRU before the classifier 1806.06228 §3.4 §3.5. The paper gives widths of 400 for the common space, 500 for the pair GRUs and 550 for the final one 1806.06228 §3.4. Card: HFusion.

Author-reported on CMU-MOSI, without context and with the paper's own features: the three-modality model reaches 77.9 accuracy, against 77.0 for early fusion and 77.3 for the second baseline 1806.06228 tab. 2. With context, the same model reaches 80.0 on MOSI, against 78.7 for the first baseline, and 76.5 accuracy on IEMOCAP, against 74.1 1806.06228 tab. 3. The paper says the context model is one to two points better than the no-context model for every combination of modalities, but it gives no table for that 1806.06228 §4.4.3. The paper states no number of runs or seeds and no spread, so the one to two point gains are not tested against run-to-run variation 1806.06228 §4.4.2.

Joint encoder over camera and LiDAR tokens

InterFuser turns each camera view and a LiDAR bird's-eye-view image into tokens. Each sensor goes through a CNN and a 1x1 convolution, and its feature map is flattened. Each token then gets a 2-D position encoding and a learned embedding for its sensor 2207.14024 §3.2 eq. 1. The inputs are four RGB views, the left, front and right cameras plus a focus crop of the front image, and the LiDAR image 2207.14024 §3.1. All tokens are concatenated and passed through one transformer encoder. Decoder queries for waypoints, a 20 by 20 object map and traffic rules read its output 2207.14024 §3.2. The token width is 256 and there are six layers in each stack 2207.14024 app. C. Card: InterFuser.

The ablation runs on the Town05 Long route with adversarial events. Concatenated inputs with no transformer fusion reach a driving score of 21.5, against 51.6 for the full model. Removing attention between views gives 46.8, and removing the sensor and position embeddings gives 48.6 2207.14024 tab. 3. Adding the LiDAR image to the front and side views raises the score from 43.0 to 49.2 2207.14024 tab. 2. Each row is a mean with a spread of several points, and the paper does not say how many runs each row averages tab. 2 tab. 3. The authors report first place on the public CARLA leaderboard, with a driving score of 76.18, but the leaderboard was accessed in June 2022 2207.14024 tab. 1. All evaluations are in the CARLA simulator 2207.14024 §5. The released code differs from the paper in the controller and the loss weights, so read the card's Paper vs code section before reimplementing it.

Early, intermediate and late fusion for 3D detection

This is a survey, not a design. It sorts LiDAR-camera fusion for 3D detection by stage: early, intermediate and late 2206.09474 §5.1. It copies the scores of each original paper, and the rows are not on one protocol. On KITTI moderate AP3D, the early-fusion row F-ConvNet scores 76.51, and the intermediate and late row CLOCs scores 80.67 2206.09474 tab. 17 tab. 18. On nuScenes, the early-fusion row PointAugmenting reports 66.8 mAP and 71.0 NDS, and the intermediate-fusion row BEVFusion reports 69.2 and 71.8 2206.09474 tab. 17 tab. 18. Card: 3D Object Detection for Autonomous Driving: A Comprehensive Survey.

The survey says that most recent fusion methods fuse in bird's-eye view, which tolerates point-to-pixel misalignment better 2206.09474 §10.1.5. It also gives a cost: PointAugmenting runs at 542 ms and CenterPoint at 70 ms on nuScenes, and the survey notes that PointAugmenting's 8.8-point mAP gain over CenterPoint comes at that cost 2206.09474 §10.1.5 tab. 17 tab. 16. Each time comes from the original paper's own hardware.

Shared code from whichever views a sample has

Each sample gets one latent code. One network per view decodes that code into the view, and the reconstruction loss counts only the views the sample has 2011.06170 §2.1 eq. 5. A second term scores each code by its mean inner product with the training codes of each class. A hinge loss on those scores gives, in the authors' argument, compact and separable clusters 2011.06170 §2.2 eq. 6 eq. 7. At test time a new sample's code is fitted to its observed views only, and the class is read from the same mean scores 2011.06170 §2.3 alg. 1. The arXiv text adds CPM-GAN, a discriminator per view trained only on the missing entries, to impute values 2011.06170 §3 eq. 10 alg. 2. Card: Deep Partial Multi-View Learning.

Author-reported CUB accuracy is 89.48%, against 86.27% for LMNN. The missing rate for that sentence is not stated 2011.06170 §4.3. At a missing rate of 0.5, with baselines filled by CRA, the gap to the second-best method is +2.5 points on CUB, +10.2 on 3Sources-complete, +36.7 on Football and +6.6 on Politics 2011.06170 §4.3 fig. 4. The text does not name the metric or say whether these are absolute or relative differences. Most other results appear only in figures 2011.06170 fig. 6 fig. 7 fig. 8. The released code has no GAN and no re-tuning step, and it differs in other ways too (see the card's Paper vs code).

Two Swin streams with alignment and an edge branch

SwinNet encodes the colour image and the second modality with two Swin-B encoders 2204.05585 §3.2. At each level, one spatial attention map, computed from the product of the two streams, weights both streams eq. 1 eq. 3. Each stream then gets its own channel weights, and these multiply its unaligned feature eq. 4 eq. 6. The two features of each level are combined by a sum and an element-wise product, then concatenated eq. 9. A decoder joins the levels from the deepest upward eq. 10. An edge branch built from the three shallowest depth levels feeds the output, and the loss is the edge loss plus the saliency loss §3.4 eq. 7 eq. 11 eq. 12. The paper describes RGB-D in detail and says RGB-T uses the same design §3.1. Card: SwinNet.

Author-reported S-measure on NLPR is 0.941 for SwinNet against 0.931 for VST, and on NJU2K 0.935 against 0.922 2204.05585 tab. 1. On the RGB-T set VT5000 it is 0.912 against 0.883 for CGFNet tab. 2. Replacing the Swin-B backbone with ResNet-50 lowers NLPR S-measure to 0.925 tab. 3. The model has 198.7 M parameters and about 124.3 GFLOPs tab. 7, and runs at about 10 FPS with I/O included §4.4.5. Training took about 26 hours on one RTX 2080 Ti §4.2. The released code differs from the equations in the spatial kernel size, the channel attention, the decoder, the loss reduction (mean, not sum) and the feature levels. It does not use the fourth Swin stage. The repository has training and test code only for RGB-D, and no LICENSE file, so the card cites its code by path only.

One transformer over image, text and lab tokens

IRENE turns a chest radiograph into 16 × 16 image patches, with a CLS token and learned positions 2306.00864 p. 10. Each chief-complaint word, each lab value, and sex and age, also get a token p. 10. Two bidirectional multimodal attention blocks let each stream attend to itself and to the other, with the cross term weighted by λ = 1.0 p. 10. The two streams are then joined and passed through ten self-attention blocks, and the averaged tokens feed the classifier p. 10 fig. 2a. Card: IRENE.

Author-reported mean AUROC over eight pulmonary diseases is 0.924, against 0.858 for Perceiver and 0.835 for early fusion 2306.00864 tab. 1. Mean AUPRC over three COVID-19 outcomes is 0.592, against 0.521 for early fusion tab. 2. Removing the radiograph drops mean AUROC to 0.543. Replacing the two bidirectional blocks with self-attention gives 0.858 tab. 3. For one COPD case the authors report 79.8% of the attention on the radiograph and 16.3% on the complaint fig. 3a. They also say that cross-attention maps cannot identify the image regions that match a keyword p. 5. The released code averages the intra-modal and inter-modal terms where the paper sums them. It uses one linear classification head where the paper uses a two-layer MLP, and it has no weights and no training loop. Its task-2 settings cannot run as configured (see the card's Paper vs code).

Cue nodes in a joint embedding for story retrieval

MovieNet is a dataset of 1,100 films with nine benchmark tasks. One task matches a plot-synopsis paragraph to a segment of a film 2007.10937 §4.5. A VSE model embeds the segment and the paragraph in one space. Extra nodes for appearance, subtitles, actions and characters are added one at a time, and the model is trained with a pairwise ranking loss and margin 0.2 §4.5 app. C.8. A graph-matching variant, MovieSynAssociation, matches character nodes with a Kuhn-Munkres assignment §4.5. Card: MovieNet.

With all added nodes, the VSE reaches R@1 18.72, R@5 44.94 and R@10 56.37, with median rank 7. MovieSynAssociation reaches R@1 21.98, and random chance is R@1 0.11 2007.10937 tab. 9. The paper's version of MovieSynAssociation uses its own features, so the comparison with the original method is not direct app. C.8. The paper is CC BY 4.0. The films, trailers, labels and features sit under the project's access terms, which describe a user agreement [project page]. Check those terms before using the data.

When to use which

Implementation sketch

Common core of the shared-and-private design, with the training loss. Each variant above changes one or two lines of it.

# Encode each modality into one vector  [arxiv:2005.03545 §3.3, eq. 1]
# Input: one feature sequence per modality; Output: one prediction
def fuse(sequences):
    h = {m: encode[m](sequences[m]) for m in modalities}  # (d,) each
    # Shared code: one encoder for all modalities  [§3.3, eq. 2]
    s = {m: shared_encoder(h[m]) for m in modalities}
    # Private code: one encoder per modality  [§3.3, eq. 2]
    p = {m: private_encoder[m](h[m]) for m in modalities}
    # Self-attention over the six codes, then concatenate  [§3.4, eq. 5]
    codes = stack([p[m] for m in modalities] + [s[m] for m in modalities])
    fused = concat(self_attention(codes))                  # (6 * d,)
    return head(fused)                                     # [§3.4]

# Training loss: task plus three weighted regularisers  [§3.5]
def loss(prediction, label, s, p, recon, h):
    total = task_loss(prediction, label)                   # [§3.5.4]
    total += alpha * similarity(s)                         # [§3.5.1]
    total += beta * difference(s, p)                       # [§3.5.2]
    total += gamma * reconstruction(recon, h)              # [§3.5.3]
    return total

Where the variants differ:

• Which codes enter fuse: both (full model), shared only, private only, or invariant only 2005.03545 tab. 4.
• Without the split, s and p are replaced by a single encoder per modality.
• The weights alpha, beta and gamma are set per dataset in the paper's table; the public code uses one set as defaults, which are the MOSI values 2005.03545 tab. 6.
• The paper writes the difference term as a squared Frobenius norm, while the public code takes a mean over the matrix. The two scale differently, so beta does not transfer between them 2005.03545 §3.5.2.

Early fusion by token concatenation, for comparison. Each stream keeps its own encoder, and only the token sequences are joined 2101.08779 §4.1.

# Early fusion by token concatenation  [arxiv:2101.08779 §4.1]
# Input: motion tokens (120, d), music tokens (240, d); Output: (20, d)
def early_fuse(motion, music):
    # Each stream has its own short transformer  [§5.2.1]
    m = motion_transformer(motion)                 # 2 layers
    a = audio_transformer(music)                   # 2 layers
    # Concatenate the motion tokens, then the music tokens  [§4.1]
    tokens = concat(m, a)                          # (360, d)
    # One deep transformer over both, full attention, no mask  [§4.1]
    out = cross_modal_transformer(tokens)          # 12 layers
    # Supervised positions: the first N = 20 outputs  [§4.1]
    return out[0 : 20]

Test-time gate, for comparison. Only the three maps are combined, and each stream is trained on its own 1907.06781 §4.2 eq. 2 eq. 3.

# Depth gate at test time  [arxiv:1907.06781 §4.2] [eq. 2] [eq. 3]
# Input: three saliency maps (H, W) in [0, 1]; Output: one map (H, W)
def depth_gate(s_rgb, s_rgbd, s_depth, t=0.15):
    # Mean absolute gap between colour-plus-depth and depth-only  [eq. 2]
    distance = mean(abs(s_rgbd - s_depth))
    # Keep the colour-plus-depth map when the two agree  [eq. 3]
    if distance <= t:
        return s_rgbd
    return s_rgb

Evidential combination of two views, for comparison. The division by one minus the conflict has no guard in the released code, so a conflict of one fails 2204.11423 §3.3 eq. 11 eq. 13.

# Evidential combination of two views  [arxiv:2204.11423 §3.2] [§3.3]
# Input: per-view evidence (K,) >= 0; Output: fused belief (K,), uncertainty
def fuse_two(e_a, e_b):
    # Dirichlet parameters and strengths per view  [§3.2] [eq. 11]
    alpha_a, alpha_b = e_a + 1, e_b + 1
    s_a, s_b = sum(alpha_a), sum(alpha_b)
    b_a, b_b = (alpha_a - 1) / s_a, (alpha_b - 1) / s_b
    u_a, u_b = K / s_a, K / s_b
    # Conflict: mass the two views put on different classes  [eq. 13]
    conflict = sum(b_a) * sum(b_b) - sum(b_a * b_b)
    # Reduced Dempster combination  [eq. 13]
    scale = 1 - conflict
    belief = (b_a * b_b + b_a * u_b + b_b * u_a) / scale
    uncertainty = u_a * u_b / scale
    return belief, uncertainty

Contradictions

The cards test different tasks with different metrics, so most of their numbers are not in conflict. The points below are where sources disagree, or where a reported number does not match a released artefact.

• Difference-loss scale (within one card). In the sentiment card, the paper and its public code disagree on the difference-loss scale, as noted above. The card records both sides and this page does not resolve it 2005.03545 §3.5.2.
• Alignment versus separation. The touch paper argues against contrastive alignment of modalities 2507.15062 §4.2. The sentiment paper uses a similarity term on shared codes 2005.03545 §3.5.1, and reports a cost when it is removed 2005.03545 tab. 4. The MMIM paper adds a contrastive term on the fusion vector 2109.00412 §3.5. STEVE-Audio aligns audio and video with a contrastive loss 2412.00949 §3.1. This is a difference in design preference across tasks, not a measured disagreement.
• Audio gains, paper's claim versus its results. STEVE-Audio's conclusion says the audio agent outperforms the text and visual agents on item-collection tasks 2412.00949 §5. Its results text reports lower ratios on seeds and cobblestone against visual, and on cobblestone against text 2412.00949 §4.1. The abstract says "comparable" [arxiv:2412.00949 abstract]. This page follows the results text.
• MISA's MOSEI MAE across cards. The MISA paper reports 0.555 for its full model 2005.03545 tab. 4. The MMIM paper lists MISA at 0.568 on MOSEI and says it took the figure from the Self-MM paper 2109.00412 tab. 2. The two papers do not give the same value, and this page does not resolve which one applies.
• Joint or separate training (within one card). The depth-gate paper describes its streams as learned jointly in its abstract. Its training section trains each stream on its own 1907.06781 §4.3. The card records the same split between wording and code.
• Reported versus reproduced numbers. TransVG reports 78.35 on RefCOCO testB with ResNet-101, and its released weights give 76.94 under the authors' own reproduction 2104.08541 tab. 2. Treat the released figure as the one an agent can expect from the public code.
• Table disagreements (within one card). The evidential card's TMC numbers for CUB and PIE differ between two tables in the same paper 2204.11423 tab. 1 tab. 2. The emotion card's CNN-only and post-check per-class numbers differ on the same emotions 2605.30099 tab. 1 tab. 2. Use the single table each claim comes from.
• ICCN's MOSEI MAE, reported versus printed. The paper's table gives 0.565 for ICCN on MOSEI 1911.05544 tab. 1. The best run printed in the released MOSEI export gives 0.5693, and that export takes its reported run on the test set 1911.05544 tab. 1. The card's Paper vs code section records both. Treat 0.565 as the paper's claim, not a reproduction.
• CenterFusion's gain, text versus table. The text gives a 35.5% relative NDS gain for the pillar and frustum steps together 2011.04841 §7. The ablation table gives +34.5% for the same row 2011.04841 tab. 3.
• DocFormer's fine-tuning epochs, supplement versus table. The supplement gives 100 epochs for FUNSD, 200 for CORD and 30 for RVL-CDIP 2106.11539 supp. §6.1. The pre-training effect table says 100 epochs for all three 2106.11539 tab. 6.
• Staged versus end-to-end training (within one card). The HFusion paper describes its method as end-to-end 1806.06228 §2. Its released code trains the pair stage first and then the trimodal stage on frozen outputs. The card records this and the paper's unstated epoch choice.
• Concatenation versus learned fusion. The sentiment position paper says that, in its authors' experience, simple concatenation performs on par with most multimodal fusion methods. It gives no measurement 2005.00357 §3.2.1. The touch paper measures concatenation at 0.50 against 0.85 for cross-attention with pretraining, on the test-tube whole task 2507.15062 tab. 1. The settings differ, and the first claim has no numbers, so this page does not resolve the disagreement.

Papers

• MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis: splits each modality into a shared and a private code, then fuses the six codes with one attention layer. Sentiment and humour results and an ablation.
• AI Choreographer: Music Conditioned 3D Dance Generation with AIST++: fuses music and motion by early token concatenation in one cross-modal transformer, with a fusion-layout ablation.
• Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality: a 400-example benchmark for whether image-text models match word-order-swapped captions to images. An evaluation, not a fusion method.
• Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper: cross-attention between image and touch embeddings, pretrained by masked tactile reconstruction, then used in a diffusion policy.
• Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks: D3Net, three streams with a test-time depth gate, a phone-captured dataset, and an 18-model benchmark.
• TransVG: End-to-End Visual Grounding with Transformers: a joint transformer over visual tokens, language tokens and a learned [REG] token that regresses one box.
• Trusted Multi-View Classification with Dynamic Evidential Fusion: per-view Dirichlet evidence fused with a reduced Dempster rule, giving a joint prediction and uncertainty.
• Evaluation of Conversational Agents: Understanding Culture, Context and Environment in Emotion Detection: face and speech classifiers with a top-three agreement check that flags sarcasm. Single run, and the paper's tables are inconsistent.
• MMIM: MMIM, mutual-information bounds on the inputs and a contrastive term on the fusion vector, added to a sentiment regression loss.
• Towards Compact Autonomous Driving Perception with Balanced Learning and Multi-sensor Fusion: one network with a shared fusion bottleneck for depth, segmentation, LiDAR segmentation and a bird's-eye-view map, with adaptive loss weights.
• An Empirical Study of Training End-to-End Vision-and-Language Transformers: METER, a detector-free image-text model with co-attention between a text and an image transformer, and a pre-training ablation study.
• ScanQA: 3D Question Answering for Spatial Scene Understanding: question answering over a 3D indoor scan, with word and object-proposal transformers and a localization head.
• ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data: a handheld gripper with a contact microphone, and a diffusion policy that fuses CLIP vision and AST audio tokens in one transformer layer.
• STEVE-Audio: Expanding the Goal Conditioning Modalities of Embodied Agents in Minecraft: audio prompts for a Minecraft agent, via a contrastive audio-video space and a conditional VAE into the agent's existing goal space.
• CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection: radar points matched to each object by a frustum and written as depth and velocity channels for second-stage heads, on a camera-only centre-point detector. Results on nuScenes and a per-class ablation.
• ICCN, the Interaction Canonical Correlation Network: outer products of the text vector with audio and video features, two 2D CNNs trained with a deep CCA loss, and an MLP on the concatenated embedding.
• DocFormer: End-to-End Transformer for Document Understanding: an encoder whose layers join text, a page-image grid and word-box features with multi-modal self-attention, pre-trained on IIT-CDIP pages.
• Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling: pairwise then trimodal fusion of text, audio and video, with a context GRU per stage. Sentiment and emotion results against early fusion.
• Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer: one transformer encoder over camera and LiDAR tokens for end-to-end driving, with an object map and a safety controller. CARLA results only.
• 3D Object Detection for Autonomous Driving: A Comprehensive Survey: a survey that sorts 3D detection methods by sensor and compiles author-reported scores. It covers LiDAR-camera fusion by stage. Not a method.
• Deep Partial Multi-View Learning: CPM-Nets, one shared code per sample from whichever views it has, with per-view reconstruction and a class loss. The GAN imputation variant is in the paper but not in the released code.
• SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detection: two Swin-B streams, with spatial and channel alignment at each level, and an edge-guided decoder.
• A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics: IRENE, one transformer over radiograph, complaint, lab and demographic tokens, with bidirectional multimodal attention blocks.
• MovieNet: A Holistic Dataset for Movie Understanding: a movie dataset with nine benchmark tasks; its synopsis retrieval adds cue nodes to a joint embedding.
• Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research: a position paper on open sentiment problems, including multimodal ones. It reports no experiments.