wikillm

Paper · 2022-06-09 · cs.LG

Does a Technique for Building Multimodal Representation Matter? -- Comparative Analysis

Compares late fusion (one model per modality, outputs merged), early fusion (concatenated input embeddings) and a hashing sketch for classification on Amazon Reviews and MovieLens. Adding modalities helped on Amazon, where late fusion with title, image and description reached 0.969 accuracy. On MovieLens25M, early fusion with text and graph reached 0.918 AUC. Author-reported.

Multimodal fusionmultimodal-learningdata-fusionrepresentation-learning

arXivPDFMarkdown11 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Many machine-learning problems come with several kinds of input at once. A product has a title, a description and a photo. A movie has a plot, a poster and a record of which users watched it. Multimodal learning turns these inputs into one representation that a model can use. This paper asks a practical question: which way of combining the inputs works best?

It compares three common ways. The late approach trains one model per input type and merges their answers at the end. The early approach joins the input vectors first and trains a single model on the joined vector. The sketch approach hashes every input into a fixed grid of mostly-zero features. It is cheap in memory and easy to extend with new inputs.

The authors test these on product classification and on movie tasks. Adding inputs helped on the product data. On the movie data the poster reduced results, so an extra input helps only when it carries information for the task. The authors say the right choice also depends on how much each input matters, the type of task, and the memory budget.

TL;DR

  • Compares late fusion, early fusion and a hashing sketch for combining text, image and graph inputs in classification.
  • Amazon Reviews: late fusion with title, image and description gives the highest accuracy, 0.969 tab. 3.
  • MovieLens25M: early fusion with text and graph gives the highest AUC, 0.918 tab. 4.
  • The sketch is the most memory-efficient option but scores lowest on accuracy §5.
  • Author-reported; each approach uses different encoders, so the comparison is not a controlled one (see Pitfalls).

Problem

  • Without a comparison it is unclear which fusion technique suits a given scenario, or which factors (modality influence, task type, memory) should decide the choice §1 §2.3.
  • Deep fusion models have many weights, need high resources and adapt poorly to changing data §2.3.1.
  • Hashing and sketch methods are cheap in memory and robust to missing modalities, but mapping high-dimensional data into a small code can lose information about single observations §2.3.2 §2.3.3.
  • The authors measure a modality's value by comparing unimodal models with multimodal ones on the same task §2.4.

Method

The paper trains three families of neural classifiers on three datasets and compares them. Section and table numbers below follow the LaTeX source of v1. The paper has no numbered equations.

  1. Encode each modality with a pretrained encoder. Text is embedded with GloVe in the late-fusion Amazon models §4.1.1 and with BERT in the early fusion and sketch models §4.1.2 §4.1.3. Images use ResNet50, with its last layer removed on Amazon §4.1.2 §4.2 §4.3. On MovieLens25M the co-view graph is embedded with Cleora (1024 dimensions) §4.2.
  2. Late fusion (Amazon only): train one model per modality, then feed their outputs into a small top network that predicts the class §4.1.1 fig. 3.
  3. Early fusion: concatenate the modality vectors at the input and train one classifier on the result §4.1.2.
  4. Sketch: map each modality into a fixed grid of depth 128 and width 512 (Amazon), either as a one-hot matrix (classical) or as a set of binary features, one per hyperplane (binarized). Flatten and concatenate the per-modality sketches, then classify §2.3.3 §4.1.3 fig. 2.
  5. MovieLens25M (multilabel genres) uses only the early-fusion and sketch approaches; the sketch has depth and width 128 §4.2.
  6. MovieLens1M (binary gender) builds a user sketch by summing the sketches of all movies that user watched, applies a width-wise L2 normalisation, and feeds the flattened result into logistic regression §4.3 fig. 5.
  7. Evaluate: accuracy on Amazon §4.1, micro-AUC and micro-mAP on MovieLens25M §4.2, and MCC on MovieLens1M because classes are imbalanced §4.3. Each setting is compared with the unimodal models that use the same approach §4 §5.

Late fusion (Amazon):

Pseudocode
# Late fusion  [§4.1.1] [fig. 3]
# Input: one embedding per modality (title, description, image)
# Output: class probabilities from a 12-unit softmax  [§4.1.1]

def late_fusion(embeddings, unimodal_models, top_model):
    # One model per modality, trained on its own input  [§4.1.1]
    outputs = [model(x) for model, x in zip(unimodal_models, embeddings)]
    # Stack the per-modality outputs into one vector  [§4.1.1]
    merged = concatenate(outputs)
    # Top network: dense 20 with ReLU, then dense 12 with softmax  [§4.1.1]
    return top_model(merged)

Early fusion (Amazon):

Pseudocode
# Early fusion  [§4.1.2]
# Input: one embedding per modality (BERT 768-d text, ResNet50 2048-d image)
# Output: class probabilities from a 12-unit softmax  [§4.1.2]

def early_fusion(embeddings):
    # Join the input vectors before any learning  [§4.1.2]
    joined = concatenate(embeddings)     # 1536, 2816 or 3584 values
    # Dense 64 with ReLU and dropout 0.1; the block count is ambiguous  [§4.1.2]
    hidden = dropout(relu(dense(joined, 64)), rate=0.1)
    # Dense 64 with ReLU, no dropout  [§4.1.2]
    hidden = relu(dense(hidden, 64))
    # Dense 12 with softmax  [§4.1.2]
    return softmax(dense(hidden, 12))

Sketch and binarized sketch (Amazon):

Pseudocode
# Sketch (classical)  [§2.3.3] [§4.1.3]
# Input: one embedding for one modality
# Output: sparse matrix, depth 128 by width 512  [§4.1.3]

def sketch(embedding, hash_functions):
    # Empty grid of depth 128 and width 512  [§4.1.3]
    grid = zeros(128, 512)
    # Assumed reading: one hash per row, marking one bucket with a 1.
    # The paper calls the sketch one-hot but does not define rows or buckets.
    for row, hash_fn in enumerate(hash_functions):
        grid[row, hash_fn(embedding)] = 1
    return grid


# Binarized sketch  [§2.3.3] [fig. 2]
# Instead of a bucket id, each hyperplane gives one binary feature  [fig. 2]

def binarized_sketch(embedding, hyperplanes):
    # Side of each hyperplane the embedding lies on; the 0/1 coding of the
    # two sides is not stated  [fig. 2]
    bits = [side_of(embedding, plane) for plane in hyperplanes]
    return bits

User representation (MovieLens1M):

Pseudocode
# User representation  [§4.3] [fig. 5]
# Input: sketches of all movies the user watched (at least 20 movies)  [§4.3]
# Output: features for logistic regression with C = 0.1  [§4.3]

def user_features(movie_sketches):
    # Sum the movie sketches into one user sketch  [§4.3]
    # Shape (210, 512) for plots, (128, 512) for posters  [§4.3]
    user = sum(movie_sketches)
    # Width-wise L2 normalisation; the axis is only shown in fig. 5  [fig. 5]
    user = l2_normalize_width(user)
    return flatten(user)

Hyperparameters

NameValueWhereNotes
Amazon: data splittrain / val / test = 0.6 / 0.2 / 0.2; test set 12,000 samples§360,000 items tab. 2
Amazon: optimizer and lossAdam; categorical cross-entropy§4.1All approaches
Amazon: batch size32§4.1Everywhere
Amazon: epochs10; binarized sketch 20§4.1
Amazon: learning rate, late fusion1e-4§4.1
Amazon: learning rate, early fusion1e-3§4.1
Amazon: learning rate, sketch1e-5§4.1Classical sketch
Amazon: learning rate, binarized sketch1e-4§4.1
Amazon: text encoder, late fusionGloVe, for title and description§4.1.1Dimension not stated
Amazon: text encoder, early fusion and sketchBERT bert-base-uncased, 768-d§4.1.2 §4.1.3Sketch section names only "BERT"
Amazon: image encoderResNet50, 2048-d with last layer removed§4.1.2 §4.1.3Late fusion: ResNet50 §4.1.1, dimension not stated
Amazon: late fusion top networkDense 20 ReLU, then dense 12 softmax§4.1.1Input 24 (bimodal), 32 (trimodal)
Amazon: early fusion input size1536 (text+text), 2816 (text+image), 3584 (trimodal)§4.1.2
Amazon: early fusion networkDense 64 ReLU + dropout 0.1; dense 64 ReLU; dense 12 softmax§4.1.2Block count ambiguous (see Pitfalls)
Amazon: sketch depth and width128 and 512§4.1.3
Amazon: sketch networkTwo blocks of dense + dropout + batch norm (1024, then 512 units); dense 128 + dropout; dense 12 softmax§4.1.3ReLU; dropout 0.2; order inside a block not stated
MovieLens25M: data60,763 movies after preprocessing; split 0.6 / 0.2 / 0.2§3 tab. 2Filtering rule not stated
MovieLens25M: graph encoderCleora, embedding size 1024, 1 iteration§4.2Cleora settings called "optimal" without a cited sweep
MovieLens25M: text encoderBERT bert-base-cased§4.2
MovieLens25M: image encoderResNet50§4.2
MovieLens25M: sketch depth and width128 and 128§4.2Amazon used width 512
MovieLens25M: networkDense 1024 ReLU, dense 512 ReLU, dense 128 ReLU, dense 20 sigmoid§4.220 outputs for multilabel genres
MovieLens25M: loss and optimizerBinary cross-entropy; Adam with default settings other than the learning rate§4.2
MovieLens25M: learning ratewritten as "1^{-5}" in the paper§4.2Read as 1e-5 by inference only
MovieLens25M: batch size64§4.2
MovieLens25M: epochs20; 30 for multimodal early fusion and graph-only unimodal models§4.2
MovieLens25M: repetitions5 runs, trained on train and validation data, scored on test§4.2Role of validation not stated
MovieLens1M: data6,040 users; split 0.8 / 0.2 train / test; k-fold CV on training set§3 tab. 2k not stated
MovieLens1M: encodersBERT bert-base-cased (plot), ResNet50 (poster)§4.3
MovieLens1M: sketch width and depthwidth 512; depth 210 (plot), 128 (poster)§4.3Classical or binarized sketch not stated
MovieLens1M: classifierLogistic regression, C = 0.1§4.3Solver and other settings not stated
MovieLens1M: metricMCC§4.3Classes imbalanced, about 71.7% male §3

Results

Author-reported. Setups are as stated above. Amazon accuracy and MovieLens AUC/mAP are given as mean ± standard deviation over the number of runs stated in each table caption. The abstract's headline numbers are 0.919 to 0.969 accuracy on Amazon and 0.907 to 0.918 AUC on MovieLens25M (abstract).

Amazon Reviews, accuracy, 10 runs tab. 3:

ModalityLate fusionEarly fusionSketchSketch (binarized)
title0.919 ± 0.0020.857 ± 0.0020.780 ± 0.0020.822 ± 0.004
description0.706 ± 0.0030.662 ± 0.0010.613 ± 0.0030.636 ± 0.004
image0.836 ± 0.0050.851 ± 0.0020.770 ± 0.0020.812 ± 0.003
title & description0.962 ± 0.0010.910 ± 0.0020.851 ± 0.0020.868 ± 0.006
image & description0.927 ± 0.0010.898 ± 0.0030.847 ± 0.0020.859 ± 0.003
image & title0.962 ± 0.0010.930 ± 0.0020.885 ± 0.0020.903 ± 0.004
image & title & description0.969 ± 0.0010.940 ± 0.0020.912 ± 0.0020.914 ± 0.002

Per-category late-fusion results are plotted in fig. 6. Bimodal and trimodal models beat unimodal ones in every category, with the largest gains in categories where unimodal models score close to each other §5.

MovieLens25M, 5 runs tab. 4:

ModalityEarly fusion AUCEarly fusion mAPSketch AUCSketch mAP
text0.907 ± 0.0010.549 ± 0.0010.877 ± 0.0030.465 ± 0.003
image0.815 ± 0.0020.350 ± 0.0010.778 ± 0.0030.297 ± 0.003
graph0.824 ± 0.0010.382 ± 0.0020.797 ± 0.0030.323 ± 0.003
text & graph0.918 ± 0.0010.592 ± 0.0020.883 ± 0.0020.484 ± 0.002
text & graph & images0.915 ± 0.0010.583 ± 0.0020.880 ± 0.0020.482 ± 0.003

MovieLens1M, gender classification on the test set, logistic regression, MCC tab. 5 (no standard deviation or number of runs is reported):

ModalityMCC
poster alone0.518
plot alone0.543
plot and poster together0.532

The paper reads these as showing that posters do not help on the MovieLens tasks §5. The authors describe the 0.543 MCC as suggestive for demographic prediction, with only simple classifiers tested §5.

Paper vs code

No official code.

The arXiv metadata lists no code link. The LaTeX source contains no repository link. A footnote that would have promised the preprocessing and source code after acceptance is commented out (raw/src/2206.06367/main.tex line 86). The paper links no project page. Searches on GitHub (repository search and user search) found nothing belonging to the authors:

  • multimodal representation sketch early fusion late fusion: unrelated hits only.
  • Pawlowski Wroblewska multimodal: no results.
  • multimodal sketch fusion: unrelated hits only.
  • user search wroblewska warsaw: no results.

Pitfalls

  • The late-fusion input sizes do not add up. The paper gives 24 for bimodal and 32 for trimodal top-network inputs §4.1.1. If each unimodal model outputs 12 values fig. 3, then bimodal gives 24 and trimodal gives 36. The paper does not reconcile this. Confirm the unimodal output size before implementing.
  • Late-fusion architectures are only partly specified. The unimodal models are based on another paper's design. The authors made small changes for the multiclass task, which concern either the loss or the layer widths, and they do not say which §4.1.1. The loss used is not stated, and the unimodal layers are not given.
  • Encoders differ across the compared approaches. Late fusion uses GloVe for text §4.1.1, while early fusion and sketch use BERT §4.1.2 §4.1.3. Differences between approaches are therefore confounded with the text encoder. Learning rates also differ by approach §4.1.
  • The early-fusion block count is ambiguous. The text says "three blocks", then describes one dense-dropout block and one dense layer without dropout §4.1.2. The card's pseudocode follows the description literally. Check the count before implementing.
  • The sketch construction is under-specified. The paper combines locality-sensitive hashing with a count-min sketch §2.3.3, but gives no hash family, number of hyperplanes or seeds. It also does not say what depth and width index, or how a binary side maps to 0 and 1 fig. 2. No code is available to resolve these.
  • Sketch sizes vary across experiments. Amazon uses depth 128 and width 512 §4.1.3. MovieLens25M uses 128 and 128 §4.2. MovieLens1M uses width 512 with depth 210 for plots and 128 for posters §4.3. The variant (classical or binarized) used for the MovieLens sketch experiments is not stated §4.2 §4.3.
  • The MovieLens25M learning rate is printed as "1^{-5}" §4.2. The intended value is probably 1e-5, but this is not confirmed.
  • The L2 normalisation in MovieLens1M appears only in a figure caption fig. 5. The text does not mention it §4.3, and the axis is not stated.
  • Missing modalities are not handled explicitly. About 25% of Amazon descriptions are missing and about 3% of titles are empty §3. On MovieLens25M, 63 movies lack a poster and 3,231 are unrated §3. The paper does not say how absent inputs are encoded (zero vector, dropped, or imputed).
  • Filtering and selection rules are missing. MovieLens25M drops movies with incorrect IMDb ids, but the criterion is not stated §3. The role of the validation split is not stated for Amazon or MovieLens25M §3 §4.2. MovieLens1M uses k-fold cross-validation without giving k §3.
  • Run counts and variance differ by table. Amazon averages 10 runs tab. 3, MovieLens25M 5 runs tab. 4, and MovieLens1M gives a single MCC per setting with no variance tab. 5. Logistic-regression settings beyond C are not given §4.3.
  • Epoch counts differ within MovieLens25M. Multimodal early fusion and graph-only unimodal models use 30 epochs; the other models use 20 §4.2. The paper does not describe early stopping.
  • Costs are qualitative. The paper says a trimodal model takes about twice the training time of a bimodal one §5, but gives no measurements. The memory benefit of the sketch is also not quantified in the results.
  • Claims beyond the data. The authors say the sketch could offer "ground-breaking perspectives" for big data, and that the MovieLens gender result could help e-commerce personalisation §5. The experiments do not test either claim.
  • The "0.919 to 0.969" headline compares within one approach. Both numbers come from the late-fusion rows of tab. 3: 0.919 is title alone, 0.969 is all three inputs. The gain is not a comparison across approaches.

Lineage

  • Builds on: the sketch comes from Dabrowski et al. (2020), an efficient manifold density estimator for recommendation, which the paper cites for the sketch's use in recommendation and similarity search §2.3.3. The late-fusion networks follow Wirojwatanakul and Wangperawong (2019) §4.1.1. Cleora (Rychalska et al., 2021) provides the graph embeddings §4.2.
  • Kiela et al. (2018), arXiv:1802.02892, is cited for the early and late fusion definitions §2.3 and among works that compare against nearest-neighbour baselines §2.4.
  • Compared against: unimodal models that use the same approach §4 §5.
  • Superseded by: not stated.

Source and license

Adapted from “Does a Technique for Building Multimodal Representation Matter? -- Comparative Analysis” by Maciej Pawłowski, Anna Wróblewska, Sylwia Sysko-Romańczuk (arXiv:2206.06367), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.