wikillm

Paper · 2022-09-07 · cs.LG, cs.AI

Multimodal learning with graphs

Perspective, no experiments. Proposes the multimodal graph learning (MGL) blueprint: identify entities, uncover topology, propagate information, mix representations. It describes graph-based multimodal designs in vision, language and natural sciences. No code and no new benchmark numbers.

Multimodal fusionGraph neural networkModality imbalancemultimodal-learninggraph-learningperspectiveblueprint

arXivPDFMarkdown6 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Graph neural networks learn from data that is already shaped like a graph, such as molecules, where atoms are linked by bonds. Many real problems mix kinds of data, like scans, notes and measurements, and the links between those kinds are often not written down anywhere. This paper is a perspective rather than a new model. It offers a four-step recipe for building such a system: decide what the units are, decide which units are connected, pass information along those connections, and combine the results into one answer. It then shows how existing work in vision, language and the natural sciences fits that recipe, and gives advice for designing new models.

TL;DR

A perspective with no experiments and no code. It proposes the multimodal graph learning (MGL) blueprint: four components (identify entities, uncover topology, propagate information, mix representations) that describe graph-based multimodal models §2.1 fig. 2. Use it to classify an existing design or to structure a new one. It gives no numbers to implement or compare; the concrete choices have to come from the works it cites.

Problem

Multimodal learning often fails in two ways. Models can rely on the most useful modality and ignore the others, a pitfall called modality collapse §1. Also, the full set of modalities is rarely available for every object, which is called missing modalities §1. Simple fusion of modalities also cannot capture the relational dependencies that different modalities create §1. The paper assumes that inductive biases differ by modality and that the graph connecting the entities is often not given explicitly [abstract] §2.1.

Method

The paper's contribution is a conceptual decomposition, not a single algorithm. Its central object is the MGL blueprint: four components applied end to end §2.1 fig. 2c. The first two (identify entities, uncover topology) form the structure learning phase, and the last two (propagate information, mix representations) form the learning on structure phase §2.1 fig. 2c. The paper also gives a background summary of graph neural networks [supp. note 1].

  1. Identify entities. Each modality i has an information matrix C_i with one vector per entity. An Identify_i module maps it into a shared namespace, giving X_i. With a single modality (k = 1) the blueprint reduces to a unimodal model §2.1 eq. 1.

    \mathbf{X}_i \leftarrow \text{Identify}_i(\mathbf{C}_i) eq. 1

  2. Uncover topology. m Connect_j modules each define one adjacency matrix A_j from a distance measure over the entities X. If the input is already an adjacency matrix, the Connect modules specify predefined neighbourhoods §2.1 eq. 2.

    \mathbf{A}_j \leftarrow \text{Connect}_j(\mathcal{X}) eq. 2

  3. Propagate information. Neural messages move along the edges of the adjacency matrices to produce node representations H. Propagation can run one independent model per adjacency matrix, or one model on a combined matrix §2.1 eq. 3.

    \mathbf{H} \leftarrow \text{Propagate}(\mathcal{X}, \mathcal{A}) eq. 3

  4. Mix representations. A Mix module turns node representations into the final entity representations Z used by the downstream task. The paper lists summation, averaging, multi-hop aggregation and methods that also read the adjacency matrices as options §2.1 eq. 4.

    \mathbf{Z} \leftarrow \text{Mix}(\mathbf{H}, \mathcal{A}) eq. 4

The four steps as a program. Names follow the blueprint equations; the paper does not define an implementation.

Pseudocode
# Component 1, identify entities  [§2.1] [eq. 1]
# Map every modality into one shared entity namespace.
entity_sets = [identify[i](C[i]) for i in range(k)]

# Component 2, uncover topology  [§2.1] [eq. 2]
# Build m adjacency matrices from m distance measures.
adjacencies = [connect[j](entity_sets) for j in range(m)]

# Component 3, propagate information  [§2.1] [eq. 3]
# Exchange messages along the edges to get node states.
node_states = propagate(entity_sets, adjacencies)

# Component 4, mix representations  [§2.1] [eq. 4]
# Turn node states into the final entity representations.
return mix(node_states, adjacencies)

Background on the GNN layer that the propagate step relies on [supp. note 1]. Each layer aggregates a node's neighbourhood (by default its 1-hop neighbours), then updates the node's own state with that aggregate, and optionally applies a final transformation. Convolutional aggregators use fixed edge weights, which follow from the graph filter (for example GCN and SGC use averaging). Message passing aggregators learn the edge importance (for example MPNN, GIN, GAT). For graph-level outputs a readout function mixes node states, for example by summation or Set2Set.

The paper then maps the blueprint onto three application areas.

  • Images (image-intensive graphs) §3 fig. 3. Nodes are regions from a segmentation such as SLIC superpixels or CNN features. Edges connect each node to its k nearest neighbours in feature space, to spatially adjacent regions, or to regions above a similarity threshold. Propagation uses graph convolution, graph attention, or residual graph networks. Human-object interaction uses two channels, one per entity type, before mixing. Visual question answering uses visual, semantic and numeric graphs with cross-channel aggregation.
  • Language (language-intensive graphs) §4 fig. 4. Graphs are built from documents, sentences, mentions, entities or words, with edges from word position, co-occurrence, or dependency parsing. For aspect-based sentiment analysis, the paper describes cooperative attention between an intra-aspect and an inter-aspect block, mixed by an interaction block and learned attention over hidden layers §4.
  • Natural sciences (knowledge-intensive graphs) §5 fig. 5. Particle jets use k-nearest-neighbour graphs. Physical simulation uses message passing on particle or joint graphs. For stereoisomers, PERM and PERM-CAT aggregation weight every permutation of a chiral group §5. QM-GNN uses global attention between atoms of two reactants to predict reactivity §5 fig. 5b. AlphaFold is described as a graph model with triangle updates §5. MaSIF uses Gaussian kernels on molecular surfaces §5 fig. 5c.

Supplementary Table S1 maps 12 methods to the four components: four image, four language and four natural science methods [supp. note 2] tab. S1. The paper's Outlook names hypergraphs, heterogeneous graphs, knowledge distillation with knowledge-intensive graphs, and visible neural networks as directions §6.

Hyperparameters

The paper reports no experiments, so no training hyperparameters exist.

NameValueWhereNotes
Experimentsnone reported§1–§6perspective; no training runs, data splits or budgets
Number of modalities knot stated§2.1 eq. 1blueprint is general in k
Number of adjacency matrices mnot stated§2.1 eq. 2set by the designer per task
Distance measures for Connect_jnot stated§2.1 eq. 2examples given per application in §3–§5
Propagation depth, hidden sizes, learning ratenot stated—not part of this paper

Results

No new results. The paper is a perspective, and its content is a classification of existing methods. Setup: 12 methods taken from the literature, grouped by application area [supp. note 2] tab. S1. The paper gives no metric values for them, so no numbers are reproduced here. Any quantitative claim about the underlying methods (for example that a graph model enabled AlphaFold's accuracy) is the authors' interpretation and not a result of this paper §5.

ItemValueSource
New experimentsnone§1–§6
Methods classified12 (4 image, 4 language, 4 natural science)tab. S1
Metrics reportednonetab. S1

Paper vs code

No official code. Searched: the arXiv metadata (no comments or code link); the paper's "Data and code availability" paragraph § Data and code availability, which points to a summary table at yashaektefaie.github.io/mgl. That page was fetched and contains no code links; it is a live list of studies, not code. A GitHub repository search for "yashaektefaie graph" returned no repositories.

Pitfalls

  • The blueprint is a framework, not an algorithm. Nothing fixes how to choose the identify, connect, propagate or mix modules, the number of adjacency matrices m, or the distance measures. An implementer must choose these, and the paper gives only examples §2.1 §3–§5.
  • Equations are cited as eq. 1–eq. 4 for the four blueprint equations in the main text. The supplement's equations are numbered separately; cite them by note [supp. note 1].
  • The box that defines the blueprint is cited as §2.1; the paper's own box number is not stated in a form that can be verified from the source.
  • The paper uses "knowledge-intensive" in the text and section headings and "knowledge-grounded" in the abstract and the Table S1 caption §5 tab. S1. Both name the same third graph type.
  • Table S1 is the authors' own classification. The source does not state its inclusion criteria tab. S1.
  • The "Data and code availability" summary is updated over time, so a page reference is not a fixed citation for the claims in this card.
  • The metadata records arXiv v6, updated 2023-01-23. The LaTeX source fetched is the one this card reflects.

Lineage

  • Builds on: graph neural network models that the paper cites as its background: MPNN (message passing, [pmlr-v70-gilmer17a]), GCN [kipf_gcn], GAT [gat], GIN [xu2018how], plus the multimodal learning surveys it cites [multimodalsurveytaxonomoy].
  • Compared against: none quantitatively. The paper's own classification of 12 methods is in tab. S1.
  • Superseded by: not stated.

Source and license

Adapted from “Multimodal learning with graphs” by Yasha Ektefaie, George Dasoulas, Ayush Noori, Maha Farhat, Marinka Zitnik (arXiv:2209.03299), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.