Manifold alignment

Overview

Manifold alignment asks that samples which are close on a graph end up close in the learned space. The graph encodes which samples should agree: nearest neighbours within one modality, and samples of the same class across modalities. The penalty is a graph-Laplacian term added to the training objective 2105.10196 §3.1.

In the one card so far, the term acts only on a shared projection. Each modality also gets its own projection with no graph term, so the alignment pulls the shared part and leaves the specific part free 2105.10196 §3.1 2105.10196 eq. 1. The card records the mechanism; it does not show that the graph term on its own accounts for the gains. The paper's ablation removes the shared/specific split and the orthogonality constraint, not the graph term 2105.10196 tab. 7.

Variants

Label-aware graph on a shared subspace

Build a graph with two kinds of edges. Within a modality, each sample links to its k nearest neighbours with a Gaussian weight. Across modalities, samples of the same class link with weight 1/N_C, where N_C is the class size 2105.10196 eq. 2 2105.10196 eq. 3 2105.10196 §3.3. The penalty is the trace of the projected data times the Laplacian, applied to the shared projection only 2105.10196 eq. 1. The paper calls this S2FL. Card: S2FL, Hong et al..

Reported results are author-reported and use a 1-nearest-neighbour classifier on the learned features. On Houston2013 (HS-MS), overall accuracy is 85.07% against 79.86% for the best baseline, l1-CoSpace 2105.10196 tab. 5. Berlin and Augsburg show the same ordering on overall accuracy 2105.10196 tab. 6 2105.10196 tab. 9.

When to use which

The card supports only one row. The other constraints are what the formulation implies, not measured results.

Implementation sketch

Pseudocode of the penalty and its graph, following 2105.10196 eq. 1 to 2105.10196 eq. 3. The shared projection is theta0; the other projections get no graph term.

# Graph W over training samples  [arxiv:2105.10196 §3.3]
# Within a modality: k-nearest-neighbour edges, Gaussian weight  [eq. 2]
# Across modalities: same-class edges, weight 1 / N_C  [eq. 3]
# Dense matrix of size 2N x 2N for two modalities  [arxiv:2105.10196 §3.2]
def build_graph(X_views, labels, k, sigma):
    W = knn_gaussian_edges(X_views, k, sigma)
    W = W + same_class_edges(labels)          # 1 / N_C per pair
    return W

# Graph Laplacian  [arxiv:2105.10196 §3.2]
def laplacian(W):
    D = diag(W.sum(axis=1))
    return D - W

# Alignment penalty on the shared projection only  [arxiv:2105.10196 eq. 1]
# theta0: (d_s, D) shared projection; X: (D, N) stacked, block-padded views
def alignment_penalty(theta0, X, L):
    Z = theta0 @ X                            # (d_s, N) shared features
    return trace(Z @ L @ Z.T)                 # weighted by beta in the loss

Contradictions

Within the single card, the paper gives two values for the graph weight on Houston2013. The text gives the optimum with β = 1 2105.10196 §4.3.1. Table 5 gives β = 0.1 2105.10196 tab. 5. The card uses 0.1 and records the conflict in its Paper vs code section 2105.10196 §4.3.1.

The official code and the paper's equations also differ on how the inter-modality weights are normalised and on the kernel width. The code gives weight 1 to every same-class pair and rescales the whole graph, while eq. 3 gives 1/N_C 2105.10196 eq. 3. The card lists these differences in full (Paper vs code). They are not conflicts between papers, but the equations alone do not reproduce the code's graph.

Papers

• S2FL, Hong et al.: S2FL, a shared and specific projection per modality, with a label-aware graph penalty on the shared part.