Dimensionality reduction

Overview

Dimensionality reduction places each high-dimensional point at a position in a lower-dimensional space, usually two dimensions for a plot, so that some structure of the data survives. The main design choice is which structure to keep. Local methods keep each point's close neighbours together. Global methods keep the overall layout, such as how far clusters sit from each other 2012.04456 §1.

The cards cover six uses so far. A 2-D visualisation method built to keep local and global structure 2012.04456 §8.1. Linear projections (PCA, and in one review also ICA and MNF) that compress features or spectral bands before a later step 2412.07823 §II.B 2103.07678 §5.1. PCA as an unsupervised check on which factors dominate a feature set 2204.06343 §4.3. A low-rank model of simulation output, dynamic mode decomposition (DMD), whose rank is chosen by error 2609.17101 §4. And feature selection, which keeps a subset of the original inputs rather than making new coordinates 2008.04103 §1.

The paper's central observation is that the usual tools favour one side. Methods that preserve local neighbourhoods well, such as t-SNE and UMAP, have trouble with global structure. TriMap does better globally and is weaker locally. Changing a single parameter does not move these methods smoothly from one behaviour to the other 2012.04456 §1 2012.04456 fig. 1 2012.04456 fig. 2.

Variants

Local-structure methods (t-SNE, UMAP)

These methods keep local neighbourhoods well. The paper reports that t-SNE is sensitive to its perplexity setting and can create spurious clusters 2012.04456 §1. No card exists for these methods yet, so nothing from their own papers is used here.

Global-structure method (TriMap)

TriMap is described as better at global layout and weaker at local neighbourhoods. The paper finds that much of its global structure comes from its PCA initialisation rather than from its triplets 2012.04456 §6 2012.04456 §3. No card exists for TriMap yet.

PaCMAP

PaCMAP builds three groups of pairs for each point: near neighbours, mid-near points and further points. It then minimises a loss on these pairs with Adam over 450 iterations in three phases, starting from a PCA layout 2012.04456 §7 2012.04456 alg. 1. The mid-near pairs are weighted heavily at first, which fixes the global layout. Their weight then drops to zero, so the neighbour pulls can refine local detail 2012.04456 §5.2. Card: Wang et al., an empirical study of dimension reduction tools and PaCMAP.

Reported results, author-reported and best of three settings per metric:

• Local datasets: PaCMAP matches local-structure methods 2012.04456 §8.2.
• Global datasets: PaCMAP matches TriMap 2012.04456 §8.2 2012.04456 fig. 21 2012.04456 fig. 22.
• SVM accuracy on COIL-20: 0.942 ± 0.009 against 0.909 for the best other method, t-SNE 2012.04456 tab. 2.
• Runtime on KDD Cup 99 (4M points): 02:05:19 against 03:34:57 for TriMap, a speed-up of 1.72 2012.04456 tab. 5.

Linear projection: PCA, ICA and MNF

PCA projects the data onto the directions of largest variance. Three cards use a linear method of this kind, and none uses one to make a 2-D map of local and global structure.

• Compression before clustering. Each row of the input matrix is one subject-task pair, built from three concatenated signals. PCA is fitted, and the first three components are kept, chosen by explained variance. Together they explain 70.04% of the variance (41.79%, 18.04% and 10.21%) 2412.07823 §II.B 2412.07823 fig. 2a. k-means then runs in that space, and K is chosen by the highest silhouette score, K = 8 with a score of 0.38 2412.07823 §III.A 2412.07823 fig. 2b. Card: An et al., optimizing locomotor task sets.
• Inspection. PCA on the feature vectors shows the main sources of variance: gripper, action and object 2204.06343 §4.3 2204.06343 fig. 8. The authors also tried t-SNE and describe it as similar, but show only PCA 2204.06343 §4.3. The paper presents this as an unsupervised check. Card: Pliska et al., single-grasp deformable object discrimination.
• Band reduction in remote sensing. A review lists PCA, MNF and ICA as ways to turn correlated bands into uncorrelated or independent components. It reports their use for lithology, alteration and structure mapping 2103.07678 §5.1. The text gives no comparison among the three 2103.07678 §5.1. The review's one numeric result from this group is a PCA-derived composite scored at 74% for lineament extraction, with the metric not defined 2103.07678 §5.1, so it cannot be set against other numbers. Card: Shirmard et al., machine learning for remote sensing in mineral exploration.

Low-rank model of simulation output: randomized DMD

Dynamic mode decomposition fits a linear model to a sequence of snapshots. Each retained mode has a Ritz value, a growth rate and a frequency 2609.17101 eq. 24. Each mode's amplitude is modelled as a power of its Ritz value over time 2609.17101 eq. 26. The randomized variant samples the range of the snapshot matrix with a random test matrix before taking a small SVD 2609.17101 alg. 2.

The number of modes is the hard part. The paper tries every rank from 2 to N_t − 1 and keeps the one with the smallest time-averaged error 2609.17101 alg. 1 2609.17101 eq. 27–29. On three viscous Burgers cases this gives 15 or 20 modes, with errors of 5.4 × 10⁻⁸ to 4.4 × 10⁻⁷ 2609.17101 tab. 1. The error is not divided by the norm of the data in the equation as printed, though the captions call it relative 2609.17101 eq. 27 2609.17101 fig. 4. Offline work takes at most about 2 s and online work 8 to 17 s, on hardware not stated 2609.17101 §6. The paper does not time the full-order solver, so no speed-up is claimed.

The paper then fits a cascade-forward network to each mode's amplitude series, forming a digital twin data model 2609.17101 §5. That network is a forecasting step, not dimensionality reduction, so it is outside the scope of this page. Card: Bistrian, randomized DMD and deep learning for digital twin data models.

Caveats from the card. All errors are in-sample: the model is evaluated on the snapshots it was built from, with no held-out time window, initial condition or Reynolds number 2609.17101 §6. The paper does not write the reduced operator in terms of the SVD factors, so an implementer must pick one 2609.17101 alg. 1. Its own correlation metric is 1.0000 for all three cases, so it does not separate them 2609.17101 tab. 1.

Feature selection: a subset, not a projection

Feature selection keeps some of the original inputs and drops the rest, so the kept features are the original measurements 2008.04103 §1. The survey groups methods into filter, wrapper, hybrid, embedded and graph-based families 2008.04103 §2. Filters score each feature with a statistic. They are cheap, but usually treat features as independent 2008.04103 §2.1 2008.04103 §5. Wrappers train a classifier for each candidate subset. They are more accurate and more expensive, especially on high-dimensional data 2008.04103 §2.2 2008.04103 §5.

The survey's experiments run one wrapper per swarm-intelligence family (11) and four filters on six UCI datasets, with SVM, Naive Bayes and AdaBoost 2008.04103 §4.3. Author-reported, the mean wrapper accuracy over the six datasets is highest for particle swarm optimisation on SVM (89.35%) and AdaBoost (88.92%) 2008.04103 §4.4. The text reports no significance test, and each family is represented by a single method, so these averages say little about the family itself 2008.04103 §4.3 2008.04103 §4.4. Card: Rostami et al., swarm intelligence feature selection review.

When to use which

The cards compare tools on 2-D embeddings only for PaCMAP, TriMap and t-SNE 2012.04456 §8.1. The PCA rows describe how the papers used PCA, not a comparison with other projections. The DMD row comes from one paper with no external baseline. The feature-selection row comes from a survey whose experiments are not a comparison with projections. The table gives the constraint, the choice the paper supports, and the evidence.

Do not drop the mid-near pairs. The paper reports that PaCMAP is not robust to omitting them, and that too many of them hurt local structure 2012.04456 §8.3 2012.04456 fig. 14.

Implementation sketch

Three sketches follow, one per card-backed method. The PaCMAP sketch is the nonlinear 2-D method, and its pairwise loss with attraction and repulsion terms is the core the paper builds on 2012.04456 §4. The PCA sketch is the linear projection used in the other cards. The DMD sketch shows only the rank search, because the reduced operator is not given in the paper. Feature selection has no sketch here, because it does not compute new coordinates; see the feature-selection card for the search loop.

# Choose neighbour pairs: nearest candidates, ranked by scaled distance,
# with sigma from the 4th to 6th neighbours  [arxiv:2012.04456 §7] [eq. 2]
neighbor_pairs = nearest_neighbor_pairs(X, n_neighbors=10)
# Draw 6 random points per point, keep the second closest; repeat
# floor(n_neighbors * 0.5) times  [arxiv:2012.04456 alg. 1]
mid_near_pairs = sample_mid_near_pairs(X, n_mid_near=5)
# Draw random non-neighbours, n_neighbors * 2 per point  [arxiv:2012.04456 §7]
further_pairs = sample_further_pairs(X, n_further=20)
# Start from a PCA layout  [arxiv:2012.04456 alg. 1]
Y = pca_initialization(X)
for step in range(1, 451):
    # Phase weights: mid-near weight falls from 1000 to 3 over steps 1-100,
    # then is 3 for steps 101-200 and 0 after  [arxiv:2012.04456 alg. 1]
    w_nb, w_mn, w_fp = phase_weights(step)
    # Loss: attraction on neighbours and mid-near pairs, repulsion on
    # further pairs; one Adam step  [arxiv:2012.04456 alg. 1]
    Y = adam_step(Y, loss_gradient(Y, w_nb, w_mn, w_fp))
return Y

# Linear projection with PCA  [arxiv:2412.07823 §II.B]
# M: one row per subject-task pair. Whether columns are also scaled is not
# stated in the paper  [arxiv:2412.07823 pitfalls]
def pca_project(M, n_components):
    # Centre the columns; PCA is defined on centred data
    centred = M - column_mean(M)
    U, S, Vt = svd(centred)
    # Share of variance per component, used to choose how many to keep
    ratio = S**2 / sum(S**2)
    # The paper keeps three components  [arxiv:2412.07823 §II.B]
    scores = centred @ Vt[:n_components].T
    return scores, ratio[:n_components]

# Rank search for randomized DMD  [arxiv:2609.17101 alg. 1] [eq. 27-29]
# V0, V1: shifted snapshot matrices; snapshots: all Nt + 1 columns
def select_dmd_rank(V0, V1, snapshots, Nt):
    best_error, best_model = infinity, None
    # Every rank from 2 to Nt - 1  [arxiv:2609.17101 alg. 1 step 1]
    for k in range(2, Nt):
        # Randomized SVD, reduced operator, eigen-decomposition: the reduced
        # operator is not specified in the paper, so it is a placeholder here
        model = rdmd_model(V0, V1, k)
        # Time-averaged error over the snapshots  [arxiv:2609.17101 eq. 28]
        error = mean_l2_error(snapshots, model.reconstruction)
        if error < best_error:
            best_error, best_model = error, model
    return best_model

Contradictions

None between cards on the same claim. Within the PaCMAP card's paper, the text says PaCMAP is "more than 1.5 times faster ... for most datasets" 2012.04456 §8.2, while its runtime table shows three speed-ups below 1.5 (USPS 1.40, MNIST 1.33, Fashion-MNIST 1.26) 2012.04456 tab. 5. The base of the speed-up ratio is not stated 2012.04456 tab. 5. The inspection card reports t-SNE as similar to PCA without showing it, so that comparison is unverified 2204.06343 §4.3.

Papers

• Rostami et al., arXiv:2008.04103: reviews swarm-based feature selection and runs one wrapper per family against four filters on six UCI datasets; a subset method, not a projection.
• Wang et al., arXiv:2012.04456: introduces PaCMAP and compares it with t-SNE, LargeVis, UMAP and TriMap on 2-D visualisation.
• Shirmard et al., arXiv:2103.07678: a review of machine learning for remote sensing in mineral exploration; reports PCA, MNF and ICA as band-reduction steps for lithology, alteration and structure mapping.
• An et al., arXiv:2412.07823: applies PCA with three components and k-means to joint-moment data, to choose representative locomotor tasks.
• Pliska et al., arXiv:2204.06343: uses PCA on squeeze features as an unsupervised check on gripper, action and object effects.
• Bistrian, arXiv:2609.17101: randomized DMD with an error-based rank search, then a neural network for the mode amplitudes; in-sample tests on three Burgers cases.