---
kind: concept
slug: manifold-alignment
title: Manifold alignment
summary: >-
  A penalty that makes a learned projection keep neighbouring samples close, using a graph built within and across views or modalities. Used to tie a shared subspace together. Only one card covers it so far, so this page describes a single formulation and its reported results.
tags: [representation-learning, multimodal, graph]
papers: ["2105.10196"]
related: [multimodal-fusion, disentangled-representation, dimensionality-reduction]
status: draft
updated_at: 2026-11-03
---

## Overview

Manifold alignment asks that samples which are close on a graph end up close
in the learned space. The graph encodes which samples should agree: nearest
neighbours within one modality, and samples of the same class across
modalities. The penalty is a graph-Laplacian term added to the training
objective [arxiv:2105.10196 §3.1].

In the one card so far, the term acts only on a shared projection. Each
modality also gets its own projection with no graph term, so the alignment
pulls the shared part and leaves the specific part free [arxiv:2105.10196 §3.1]
[arxiv:2105.10196 eq. 1]. The card records the mechanism; it does not show
that the graph term on its own accounts for the gains. The paper's ablation
removes the shared/specific split and the orthogonality constraint, not the
graph term [arxiv:2105.10196 tab. 7].

## Variants

### Label-aware graph on a shared subspace

Build a graph with two kinds of edges. Within a modality, each sample links to
its k nearest neighbours with a Gaussian weight. Across modalities, samples of
the same class link with weight 1/N_C, where N_C is the class size
[arxiv:2105.10196 eq. 2] [arxiv:2105.10196 eq. 3] [arxiv:2105.10196 §3.3].
The penalty is the trace of the projected data times the Laplacian, applied
to the shared projection only [arxiv:2105.10196 eq. 1]. The paper calls this
S2FL. Card: [S2FL, Hong et al.](/papers/2105.10196).

Reported results are author-reported and use a 1-nearest-neighbour classifier
on the learned features. On Houston2013 (HS-MS), overall accuracy is 85.07%
against 79.86% for the best baseline, l1-CoSpace [arxiv:2105.10196 tab. 5].
Berlin and Augsburg show the same ordering on overall accuracy
[arxiv:2105.10196 tab. 6] [arxiv:2105.10196 tab. 9].

## When to use which

The card supports only one row. The other constraints are what the formulation
implies, not measured results.

| Situation | Use | Why |
|---|---|---|
| Several views of the same samples, labels for the training set, and a sample count where a dense 2N×2N Laplacian fits | This variant | The inter-modality edges need class labels [arxiv:2105.10196 eq. 3]; the paper's experiments are in this regime [arxiv:2105.10196 §4.1] |
| Unlabelled data or labels that are unreliable | Not covered by this card | The inter-modality weights are defined per class [arxiv:2105.10196 eq. 3]; no unlabelled setting is tested |
| Large sample counts | Check memory first | The official demo builds a dense 2N×2N matrix [arxiv:2105.10196 §3.3]; the card's Pitfalls section gives the arithmetic. The paper does not discuss memory |
| Downstream model other than nearest neighbour | Not tested | The paper chose 1-NN on purpose so that feature quality is measured [arxiv:2105.10196 §4.2.1] |

## Implementation sketch

Pseudocode of the penalty and its graph, following [arxiv:2105.10196 eq. 1]
to [arxiv:2105.10196 eq. 3]. The shared projection is theta0; the other
projections get no graph term.

```pseudocode
# Graph W over training samples  [arxiv:2105.10196 §3.3]
# Within a modality: k-nearest-neighbour edges, Gaussian weight  [eq. 2]
# Across modalities: same-class edges, weight 1 / N_C  [eq. 3]
# Dense matrix of size 2N x 2N for two modalities  [arxiv:2105.10196 §3.2]
def build_graph(X_views, labels, k, sigma):
    W = knn_gaussian_edges(X_views, k, sigma)
    W = W + same_class_edges(labels)          # 1 / N_C per pair
    return W

# Graph Laplacian  [arxiv:2105.10196 §3.2]
def laplacian(W):
    D = diag(W.sum(axis=1))
    return D - W

# Alignment penalty on the shared projection only  [arxiv:2105.10196 eq. 1]
# theta0: (d_s, D) shared projection; X: (D, N) stacked, block-padded views
def alignment_penalty(theta0, X, L):
    Z = theta0 @ X                            # (d_s, N) shared features
    return trace(Z @ L @ Z.T)                 # weighted by beta in the loss
```

## Contradictions

Within the single card, the paper gives two values for the graph weight on
Houston2013. The text gives the optimum with β = 1 [arxiv:2105.10196 §4.3.1].
Table 5 gives β = 0.1 [arxiv:2105.10196 tab. 5]. The card uses 0.1 and records
the conflict in its Paper vs code section [arxiv:2105.10196 §4.3.1].

The official code and the paper's equations also differ on how the
inter-modality weights are normalised and on the kernel width. The code gives
weight 1 to every same-class pair and rescales the whole graph, while eq. 3
gives 1/N_C [arxiv:2105.10196 eq. 3]. The card lists these differences in full
([Paper vs code](/papers/2105.10196)). They are not conflicts between papers,
but the equations alone do not reproduce the code's graph.

## Papers

- [S2FL, Hong et al.](/papers/2105.10196): S2FL, a shared and specific projection per modality, with a label-aware graph penalty on the shared part.