--- kind: concept slug: dimensionality-reduction title: Dimensionality reduction summary: >- Maps high-dimensional points to fewer dimensions, usually 2-D for plots, keeping their structure. Methods trade local neighbourhoods against global layout. PaCMAP keeps both and is reported faster than TriMap on large data. PCA, ICA and MNF are linear options. Feature selection keeps original inputs instead. tags: [visualization, manifold-learning, embedding, reduced-order-modeling] papers: ["2008.04103", "2012.04456", "2103.07678", "2204.06343", "2412.07823", "2609.17101"] related: [nearest-neighbor-search, intrinsic-dimension, spectral-methods, k-means-clustering, digital-twin, surrogate-modeling, feature-selection, dynamic-mode-decomposition] status: draft updated_at: 2026-11-01 --- ## Overview Dimensionality reduction places each high-dimensional point at a position in a lower-dimensional space, usually two dimensions for a plot, so that some structure of the data survives. The main design choice is which structure to keep. Local methods keep each point's close neighbours together. Global methods keep the overall layout, such as how far clusters sit from each other [arxiv:2012.04456 §1]. The cards cover six uses so far. A 2-D visualisation method built to keep local and global structure [arxiv:2012.04456 §8.1]. Linear projections (PCA, and in one review also ICA and MNF) that compress features or spectral bands before a later step [arxiv:2412.07823 §II.B] [arxiv:2103.07678 §5.1]. PCA as an unsupervised check on which factors dominate a feature set [arxiv:2204.06343 §4.3]. A low-rank model of simulation output, dynamic mode decomposition (DMD), whose rank is chosen by error [arxiv:2609.17101 §4]. And feature selection, which keeps a subset of the original inputs rather than making new coordinates [arxiv:2008.04103 §1]. The paper's central observation is that the usual tools favour one side. Methods that preserve local neighbourhoods well, such as t-SNE and UMAP, have trouble with global structure. TriMap does better globally and is weaker locally. Changing a single parameter does not move these methods smoothly from one behaviour to the other [arxiv:2012.04456 §1] [arxiv:2012.04456 fig. 1] [arxiv:2012.04456 fig. 2]. ## Variants ### Local-structure methods (t-SNE, UMAP) These methods keep local neighbourhoods well. The paper reports that t-SNE is sensitive to its perplexity setting and can create spurious clusters [arxiv:2012.04456 §1]. No card exists for these methods yet, so nothing from their own papers is used here. ### Global-structure method (TriMap) TriMap is described as better at global layout and weaker at local neighbourhoods. The paper finds that much of its global structure comes from its PCA initialisation rather than from its triplets [arxiv:2012.04456 §6] [arxiv:2012.04456 §3]. No card exists for TriMap yet. ### PaCMAP PaCMAP builds three groups of pairs for each point: near neighbours, mid-near points and further points. It then minimises a loss on these pairs with Adam over 450 iterations in three phases, starting from a PCA layout [arxiv:2012.04456 §7] [arxiv:2012.04456 alg. 1]. The mid-near pairs are weighted heavily at first, which fixes the global layout. Their weight then drops to zero, so the neighbour pulls can refine local detail [arxiv:2012.04456 §5.2]. Card: [Wang et al., an empirical study of dimension reduction tools and PaCMAP](/papers/2012.04456). Reported results, author-reported and best of three settings per metric: - Local datasets: PaCMAP matches local-structure methods [arxiv:2012.04456 §8.2]. - Global datasets: PaCMAP matches TriMap [arxiv:2012.04456 §8.2] [arxiv:2012.04456 fig. 21] [arxiv:2012.04456 fig. 22]. - SVM accuracy on COIL-20: 0.942 ± 0.009 against 0.909 for the best other method, t-SNE [arxiv:2012.04456 tab. 2]. - Runtime on KDD Cup 99 (4M points): 02:05:19 against 03:34:57 for TriMap, a speed-up of 1.72 [arxiv:2012.04456 tab. 5]. ### Linear projection: PCA, ICA and MNF PCA projects the data onto the directions of largest variance. Three cards use a linear method of this kind, and none uses one to make a 2-D map of local and global structure. - **Compression before clustering.** Each row of the input matrix is one subject-task pair, built from three concatenated signals. PCA is fitted, and the first three components are kept, chosen by explained variance. Together they explain 70.04% of the variance (41.79%, 18.04% and 10.21%) [arxiv:2412.07823 §II.B] [arxiv:2412.07823 fig. 2a]. k-means then runs in that space, and K is chosen by the highest silhouette score, K = 8 with a score of 0.38 [arxiv:2412.07823 §III.A] [arxiv:2412.07823 fig. 2b]. Card: [An et al., optimizing locomotor task sets](/papers/2412.07823). - **Inspection.** PCA on the feature vectors shows the main sources of variance: gripper, action and object [arxiv:2204.06343 §4.3] [arxiv:2204.06343 fig. 8]. The authors also tried t-SNE and describe it as similar, but show only PCA [arxiv:2204.06343 §4.3]. The paper presents this as an unsupervised check. Card: [Pliska et al., single-grasp deformable object discrimination](/papers/2204.06343). - **Band reduction in remote sensing.** A review lists PCA, MNF and ICA as ways to turn correlated bands into uncorrelated or independent components. It reports their use for lithology, alteration and structure mapping [arxiv:2103.07678 §5.1]. The text gives no comparison among the three [arxiv:2103.07678 §5.1]. The review's one numeric result from this group is a PCA-derived composite scored at 74% for lineament extraction, with the metric not defined [arxiv:2103.07678 §5.1], so it cannot be set against other numbers. Card: [Shirmard et al., machine learning for remote sensing in mineral exploration](/papers/2103.07678). ### Low-rank model of simulation output: randomized DMD Dynamic mode decomposition fits a linear model to a sequence of snapshots. Each retained mode has a Ritz value, a growth rate and a frequency [arxiv:2609.17101 eq. 24]. Each mode's amplitude is modelled as a power of its Ritz value over time [arxiv:2609.17101 eq. 26]. The randomized variant samples the range of the snapshot matrix with a random test matrix before taking a small SVD [arxiv:2609.17101 alg. 2]. The number of modes is the hard part. The paper tries every rank from 2 to N_t − 1 and keeps the one with the smallest time-averaged error [arxiv:2609.17101 alg. 1] [arxiv:2609.17101 eq. 27–29]. On three viscous Burgers cases this gives 15 or 20 modes, with errors of 5.4 × 10⁻⁸ to 4.4 × 10⁻⁷ [arxiv:2609.17101 tab. 1]. The error is not divided by the norm of the data in the equation as printed, though the captions call it relative [arxiv:2609.17101 eq. 27] [arxiv:2609.17101 fig. 4]. Offline work takes at most about 2 s and online work 8 to 17 s, on hardware not stated [arxiv:2609.17101 §6]. The paper does not time the full-order solver, so no speed-up is claimed. The paper then fits a cascade-forward network to each mode's amplitude series, forming a digital twin data model [arxiv:2609.17101 §5]. That network is a forecasting step, not dimensionality reduction, so it is outside the scope of this page. Card: [Bistrian, randomized DMD and deep learning for digital twin data models](/papers/2609.17101). Caveats from the card. All errors are in-sample: the model is evaluated on the snapshots it was built from, with no held-out time window, initial condition or Reynolds number [arxiv:2609.17101 §6]. The paper does not write the reduced operator in terms of the SVD factors, so an implementer must pick one [arxiv:2609.17101 alg. 1]. Its own correlation metric is 1.0000 for all three cases, so it does not separate them [arxiv:2609.17101 tab. 1]. ### Feature selection: a subset, not a projection Feature selection keeps some of the original inputs and drops the rest, so the kept features are the original measurements [arxiv:2008.04103 §1]. The survey groups methods into filter, wrapper, hybrid, embedded and graph-based families [arxiv:2008.04103 §2]. Filters score each feature with a statistic. They are cheap, but usually treat features as independent [arxiv:2008.04103 §2.1] [arxiv:2008.04103 §5]. Wrappers train a classifier for each candidate subset. They are more accurate and more expensive, especially on high-dimensional data [arxiv:2008.04103 §2.2] [arxiv:2008.04103 §5]. The survey's experiments run one wrapper per swarm-intelligence family (11) and four filters on six UCI datasets, with SVM, Naive Bayes and AdaBoost [arxiv:2008.04103 §4.3]. Author-reported, the mean wrapper accuracy over the six datasets is highest for particle swarm optimisation on SVM (89.35%) and AdaBoost (88.92%) [arxiv:2008.04103 §4.4]. The text reports no significance test, and each family is represented by a single method, so these averages say little about the family itself [arxiv:2008.04103 §4.3] [arxiv:2008.04103 §4.4]. Card: [Rostami et al., swarm intelligence feature selection review](/papers/2008.04103). ## When to use which The cards compare tools on 2-D embeddings only for PaCMAP, TriMap and t-SNE [arxiv:2012.04456 §8.1]. The PCA rows describe how the papers used PCA, not a comparison with other projections. The DMD row comes from one paper with no external baseline. The feature-selection row comes from a survey whose experiments are not a comparison with projections. The table gives the constraint, the choice the paper supports, and the evidence. | Situation | Use | Why | |---|---|---| | A 2-D plot where both cluster separation and local neighbourhoods matter | PaCMAP | Matches local methods on local datasets and TriMap on global ones [arxiv:2012.04456 §8.2] | | Only local neighbourhoods matter, and the global layout is not read | t-SNE or UMAP | The paper reports these as strong locally; no card covers them, so their numbers are not used here [arxiv:2012.04456 §1] | | Only the global layout matters | TriMap, or PaCMAP | TriMap is the paper's global reference; PaCMAP matches it on global datasets [arxiv:2012.04456 §8.2] | | Large data, with a runtime budget | PaCMAP | Speed-ups in the table range from 1.26 to 2.23 on one server; against TriMap, 1.72 on KDD Cup 99 (4M) [arxiv:2012.04456 tab. 5] | | Fewer features before clustering, with a stated share of variance | PCA, then k-means | Three components explain 70.04%, and K is picked by silhouette; no other projection is compared [arxiv:2412.07823 §II.B] [arxiv:2412.07823 §III.A] | | Finding which factors dominate the variance of a feature set, without labels | PCA | Used as the check in the paper, which shows only PCA; t-SNE is called similar without a figure [arxiv:2204.06343 §4.3] | | Correlated spectral bands in remote sensing, before mapping lithology, alteration or structures | PCA, ICA or MNF | Reported as used for these targets; the text gives no comparison among the three [arxiv:2103.07678 §5.1] | | Output must be a subset of the original inputs | Feature selection: filter for speed, wrapper for accuracy | Filters are cheap but usually treat features as independent; wrappers fit a classifier per candidate subset [arxiv:2008.04103 §2.1] [arxiv:2008.04103 §2.2] | | Scarce data with many features | Reduce the feature count first | The survey's general rule is at least 10 × n × C training examples, for n features and C classes [arxiv:2008.04103 §2] | | Modes with growth rates and frequencies from time-resolved simulation data | Randomized DMD with error-based rank search | Rank chosen by time-averaged error; the reduced operator must be chosen by the implementer [arxiv:2609.17101 alg. 1] | | Accuracy on unseen times, initial conditions or settings | Not shown by this card; validate first | Every reported error is in-sample on three cases [arxiv:2609.17101 §6] | | No time to tune, or a single fixed setting | Expect lower numbers than reported | Reported scores use the best of three settings per metric [arxiv:2012.04456 §8.1] | | Layout must not depend on the starting positions | PaCMAP, with care | The paper argues its layout does not depend on initialisation, but this is shown qualitatively [arxiv:2012.04456 §8.3] | | Need to match the paper's results exactly | Code defaults, documented | The learning rate and Adam constants are not stated in the paper [arxiv:2012.04456 alg. 1] | Do not drop the mid-near pairs. The paper reports that PaCMAP is not robust to omitting them, and that too many of them hurt local structure [arxiv:2012.04456 §8.3] [arxiv:2012.04456 fig. 14]. ## Implementation sketch Three sketches follow, one per card-backed method. The PaCMAP sketch is the nonlinear 2-D method, and its pairwise loss with attraction and repulsion terms is the core the paper builds on [arxiv:2012.04456 §4]. The PCA sketch is the linear projection used in the other cards. The DMD sketch shows only the rank search, because the reduced operator is not given in the paper. Feature selection has no sketch here, because it does not compute new coordinates; see the feature-selection card for the search loop. ```pseudocode # Choose neighbour pairs: nearest candidates, ranked by scaled distance, # with sigma from the 4th to 6th neighbours [arxiv:2012.04456 §7] [eq. 2] neighbor_pairs = nearest_neighbor_pairs(X, n_neighbors=10) # Draw 6 random points per point, keep the second closest; repeat # floor(n_neighbors * 0.5) times [arxiv:2012.04456 alg. 1] mid_near_pairs = sample_mid_near_pairs(X, n_mid_near=5) # Draw random non-neighbours, n_neighbors * 2 per point [arxiv:2012.04456 §7] further_pairs = sample_further_pairs(X, n_further=20) # Start from a PCA layout [arxiv:2012.04456 alg. 1] Y = pca_initialization(X) for step in range(1, 451): # Phase weights: mid-near weight falls from 1000 to 3 over steps 1-100, # then is 3 for steps 101-200 and 0 after [arxiv:2012.04456 alg. 1] w_nb, w_mn, w_fp = phase_weights(step) # Loss: attraction on neighbours and mid-near pairs, repulsion on # further pairs; one Adam step [arxiv:2012.04456 alg. 1] Y = adam_step(Y, loss_gradient(Y, w_nb, w_mn, w_fp)) return Y ``` ```pseudocode # Linear projection with PCA [arxiv:2412.07823 §II.B] # M: one row per subject-task pair. Whether columns are also scaled is not # stated in the paper [arxiv:2412.07823 pitfalls] def pca_project(M, n_components): # Centre the columns; PCA is defined on centred data centred = M - column_mean(M) U, S, Vt = svd(centred) # Share of variance per component, used to choose how many to keep ratio = S**2 / sum(S**2) # The paper keeps three components [arxiv:2412.07823 §II.B] scores = centred @ Vt[:n_components].T return scores, ratio[:n_components] ``` ```pseudocode # Rank search for randomized DMD [arxiv:2609.17101 alg. 1] [eq. 27-29] # V0, V1: shifted snapshot matrices; snapshots: all Nt + 1 columns def select_dmd_rank(V0, V1, snapshots, Nt): best_error, best_model = infinity, None # Every rank from 2 to Nt - 1 [arxiv:2609.17101 alg. 1 step 1] for k in range(2, Nt): # Randomized SVD, reduced operator, eigen-decomposition: the reduced # operator is not specified in the paper, so it is a placeholder here model = rdmd_model(V0, V1, k) # Time-averaged error over the snapshots [arxiv:2609.17101 eq. 28] error = mean_l2_error(snapshots, model.reconstruction) if error < best_error: best_error, best_model = error, model return best_model ``` ## Contradictions None between cards on the same claim. Within the PaCMAP card's paper, the text says PaCMAP is "more than 1.5 times faster ... for most datasets" [arxiv:2012.04456 §8.2], while its runtime table shows three speed-ups below 1.5 (USPS 1.40, MNIST 1.33, Fashion-MNIST 1.26) [arxiv:2012.04456 tab. 5]. The base of the speed-up ratio is not stated [arxiv:2012.04456 tab. 5]. The inspection card reports t-SNE as similar to PCA without showing it, so that comparison is unverified [arxiv:2204.06343 §4.3]. ## Papers - [Rostami et al., arXiv:2008.04103](/papers/2008.04103): reviews swarm-based feature selection and runs one wrapper per family against four filters on six UCI datasets; a subset method, not a projection. - [Wang et al., arXiv:2012.04456](/papers/2012.04456): introduces PaCMAP and compares it with t-SNE, LargeVis, UMAP and TriMap on 2-D visualisation. - [Shirmard et al., arXiv:2103.07678](/papers/2103.07678): a review of machine learning for remote sensing in mineral exploration; reports PCA, MNF and ICA as band-reduction steps for lithology, alteration and structure mapping. - [An et al., arXiv:2412.07823](/papers/2412.07823): applies PCA with three components and k-means to joint-moment data, to choose representative locomotor tasks. - [Pliska et al., arXiv:2204.06343](/papers/2204.06343): uses PCA on squeeze features as an unsupervised check on gripper, action and object effects. - [Bistrian, arXiv:2609.17101](/papers/2609.17101): randomized DMD with an error-based rank search, then a neural network for the mode amplitudes; in-sample tests on three Burgers cases.