--- kind: concept slug: feature-selection title: Feature selection summary: >- Keeping a subset of candidate input features: filter, wrapper (including swarm search), embedded (L1) and safe screening, which drops Lasso features that are provably zero before solving. Exhaustive search grows as 2^n. Reported speed-ups and accuracies are author-reported on different data and are not comparable. tags: [feature-selection, wrapper-methods, embedded-methods, safe-screening, lasso, univariate-ranking, recursive-feature-elimination, swarm-intelligence, class-imbalance, review] papers: ["2301.08826", "1211.3966", "2406.13547", "1305.1707", "2605.04666", "2609.07408", "2008.04103", "2012.05999", "2407.03646"] related: [adaboost, feature-scaling, random-forest, support-vector-machine, intrusion-detection, lasso-regression, interpretable-machine-learning, canny-edge-detection, ensemble-learning, metaheuristic-optimization, particle-swarm-optimization, ai-generated-text-detection, dimensionality-reduction] status: draft updated_at: 2026-11-01 --- ## Overview Feature selection keeps some of the candidate features and drops the rest. The review places it after feature extraction, which turns text into vectors such as bag-of-words, TF-IDF or word embeddings [arxiv:2301.08826 §2.1]. The invariant it states is the size of the search space: n features give 2^n possible subsets [arxiv:2301.08826 §2.2]. Nine cards on this wiki list the concept, each from a different setting. The review gives the three families but runs no experiments [arxiv:2301.08826 §5]. A safe-screening paper gives exact rules that drop Lasso features before the Lasso is solved [arxiv:1211.3966 §2.3.3]. A web-firewall paper uses L1 regularisation to set some rule weights to zero, which selects rules [arxiv:2406.13547 §4.4]. A clinical paper ranks temporal features one at a time by AUC and trains a linear SVM on the top few [arxiv:2605.04666 p. 3]. A medical-imaging paper selects image statistics with recursive feature elimination (RFE) [arxiv:2609.07408 p. 10]. A class-imbalance review lists feature selection as one of three families of remedies [arxiv:1305.1707 §2.3]. A swarm-intelligence survey runs wrapper and filter methods on six UCI datasets [arxiv:2008.04103 §4]. A heart-disease paper uses a metaheuristic search to pick attributes [arxiv:2012.05999 §II-B]. A stylometry study keeps the features that a LASSO model gives non-zero weight [arxiv:2407.03646 §2.2]. ## Variants ### Filter methods Filter methods score features without training the final model. The review names the family [arxiv:2301.08826 §2.2] but gives no procedure for it. Chi-square and information gain appear only as the tools used in cited studies [arxiv:2301.08826 §2.2.3]. Card: [Shaik et al., 2023](/papers/2301.08826). The class-imbalance review suggests filters for high-dimensional data: each feature is scored on its own, and the user picks the number j of features to keep [arxiv:1305.1707 §2.3]. It gives no value for j and no criterion beyond the per-feature scores. Card: [Longadge and Dongre, 2013](/papers/1305.1707). The clinical paper ranks each feature by its single-feature AUC for one prediction target, with no joint search [arxiv:2605.04666 p. 3]. It then keeps the top 1, 3 or 30 features [arxiv:2605.04666 tab. 1] [arxiv:2605.04666 tab. 2]. The ranking is cheap because it scores each feature alone. The authors say that with more than 9,000 candidates this greedy step may not find the best set [arxiv:2605.04666 p. 5]. Card: [Valko and Hauskrecht, 2026](/papers/2605.04666). The swarm survey runs four filter methods, one per study it chose to represent the filter side, and reports their mean accuracy over six datasets. On SVM the best filter averages 79.52%, from an ant-colony method, and the wrapper average for the same classifier is higher [arxiv:2008.04103 §4.4]. The survey also states that filters usually treat features as independent and tend to reach lower-quality subsets [arxiv:2008.04103 §2.1]. Card: [Rostami et al., 2020](/papers/2008.04103). ### Wrapper methods Wrapper methods search over subsets and score each one with a model. The review names sequential forward selection as an example [arxiv:2301.08826 §2.2.2]. It adds one feature at a time, so it visits far fewer subsets than the 2^n total. Card: [Shaik et al., 2023](/papers/2301.08826). The medical-imaging paper uses RFE to rank the eleven image statistics, separately for each model, and keeps the features ranked 2 or better [arxiv:2609.07408 alg. 3] [arxiv:2609.07408 p. 10]. The paper does not state the estimator, step size or fold count used by RFE [arxiv:2609.07408 alg. 3]. The feature sets it uses in the experiments were then chosen by reading correlation matrices, with no stated rule [arxiv:2609.07408 p. 13] [arxiv:2609.07408 p. 22]. Card: [Bhat and Szczuko, 2026](/papers/2609.07408). Population-based (swarm) search is another way to explore subsets. The swarm survey evaluates one wrapper per swarm family, eleven in all, and reports for each the mean over six UCI datasets with three classifiers [arxiv:2008.04103 §3] [arxiv:2008.04103 §4.3]. On SVM and AdaBoost, particle swarm optimisation (PSO) has the highest mean. On Naive Bayes, the cuckoo method is first and PSO second, a gap of 0.16 points [arxiv:2008.04103 §4.4]. The authors say PSO does better than the other wrappers in most cases, but they run no significance test [arxiv:2008.04103 §4.4]. Its wrappers kept 7.21% of Arrhythmia's features and 0.58% of Colon's [arxiv:2008.04103 tab. 5]. The survey states no population size, iteration count, stopping rule or fitness function for any method [arxiv:2008.04103 §4]. Card: [Rostami et al., 2020](/papers/2008.04103). The heart-disease paper uses a modified cuttlefish search to pick attributes before its classifier is tuned [arxiv:2012.05999 §II-B]. It selects 7 of the 13 UCI attributes, while the compared methods select between 6 and 10, or keep all 13 [arxiv:2012.05999 tab. 6]. The fitness function of the search and its population size are not stated [arxiv:2012.05999 §II-E]. Card: [Khan, 2020](/papers/2012.05999). ### Embedded methods Embedded methods select features as part of training. The review gives L1 regularisation as an example and reports random-forest importance from a cited study [arxiv:2301.08826 §2.2.3]. An L1 penalty can set some weights to exactly zero, and the features with zero weight are then dropped. ModSec-Learn trains linear SVMs, logistic regression (LR) and random forests on binary features, one per SQL-injection rule of the OWASP Core Rule Set, which either fired or did not [arxiv:2406.13547 §3]. At paranoia level 4, LR with L1 gives weight 0 to 18 rules, while the L2 model and ModSecurity use all of them [arxiv:2406.13547 §4.4]. LR with L1 reaches 99.35% TPR at 1% FPR there, against 99.34% for LR with L2 [arxiv:2406.13547 tab. 1]. These are the authors' numbers on their balanced test set. The 18-rule count has not been checked against the released model. Card: [Scano et al., 2024](/papers/2406.13547). The stylometry study fits one binomial LASSO per linguistic domain, with the AI-or-human label as outcome, and takes the features with non-zero coefficients at the penalty that minimises cross-validated error [arxiv:2407.03646 §2.2]. Across the four domains it keeps 13 features, including primary stress, alveolar consonants and pronouns [arxiv:2407.03646 tab. 3]. The features are not scaled before fitting, so the coefficient sizes cannot be compared [arxiv:2407.03646 §2.2]. The study has ten texts in all, and the card notes that the penalty, folds and seed are not stated, so a rerun on another fold split may keep different features [arxiv:2407.03646 p. 6]. Card: [Georgiou, 2024](/papers/2407.03646). ### Safe screening for the Lasso Safe screening is not a search over models. Before the Lasso is solved, it drops features whose coefficients are guaranteed to be zero, so the smaller problem gives the same answer as the full one [arxiv:1211.3966 §1]. The enhanced rule (EDPP) tests each feature against a small region around the dual solution at a nearby penalty [arxiv:1211.3966 §2.3.3]. Along a decreasing grid of penalties, each test uses the solution at the previous penalty [arxiv:1211.3966 §2.2]. Any Lasso solver can run after the screen [arxiv:1211.3966 §1]. The rules are derived for one scaling of the Lasso objective, so other conventions must be mapped first [arxiv:1211.3966 §2.1]. The authors report these solver times for a 100-penalty path with the SLEP solver. On Prostate the solver alone takes 121.41 s and EDPP plus solver takes 3.70 s. On PIE the figures are 629.94 s and 4.13 s, and on MNIST 2566.26 s and 11.12 s [arxiv:1211.3966 tab. 1]. The paper also extends the rules to the group Lasso [arxiv:1211.3966 §3]. Card: [Wang et al., 2012](/papers/1211.3966). ### Reported results The review quotes these accuracies from cited studies [arxiv:2301.08826 §2.2.3]. Each uses its own dataset and baselines, so they are not comparable. | Study (as cited in the review) | Selection method | Reported result | |---|---|---| | Gutiérrez et al. | random-forest importance, SVM with radial kernel | 85.17% accuracy | | Pong-Inwong et al. | chi-square, vote ensemble | 87.16% accuracy | | Soukaina et al. | information gain | SVM 85.9% after selection, random forest 81.6% before | The cards on this wiki report these results, each on its own data and setup. The clinical paper's linear SVM uses the top-k features from univariate AUC ranking [arxiv:2605.04666 tab. 1] [arxiv:2605.04666 tab. 2]. Its numbers are AUCs for one hospital's records, with no external baseline. | Paper | Selection method | Reported result | |---|---|---| | [Valko and Hauskrecht, 2026](/papers/2605.04666) | univariate AUC, top-1 vs top-30, linear SVM | 10 medications: top-1 AUC 69.62% to 87.65%; top-30 AUC 80.53% to 89.75% [arxiv:2605.04666 tab. 2] | | [Bhat and Szczuko, 2026](/papers/2609.07408) | RFE, then per-dataset feature sets | DS2 (augmented, no edge step) AUC 0.97 for RF, XGBoost and KNN [arxiv:2609.07408 fig. 13] | | [Rostami et al., 2020](/papers/2008.04103) | 11 swarm wrappers, mean over six UCI sets | SVM: PSO 89.35% first, ACO 88.85%; Naive Bayes: COA 89.24% first, PSO 89.08%; AdaBoost: PSO 88.92% first [arxiv:2008.04103 §4.4] | | [Rostami et al., 2020](/papers/2008.04103) | 4 filter methods, mean over six UCI sets | SVM: ACO 79.52% first, PSO 78.82% [arxiv:2008.04103 §4.4] | | [Khan, 2020](/papers/2012.05999) | modified cuttlefish search, 7 of 13 attributes, then MDCNN | Framingham (4000 records) 98.2% accuracy vs 88.3% for LR and 83.8% for a deep network [arxiv:2012.05999 tab. 8] | | [Georgiou, 2024](/papers/2407.03646) | per-domain LASSO on 35 tested measures | 13 non-zero coefficients; the card records no classification accuracy for this model [arxiv:2407.03646 tab. 3] | The Bhat and Szczuko card records no run with all eleven statistics, so its numbers do not isolate the effect of selection. Its RFE was also run before cross-validation, which the card flags as optimistic [arxiv:2609.07408 alg. 3]. The Khan card records no run that uses all 13 attributes with the same classifier, so its 98.2% cannot be credited to the selection step. The paper also names no split, and reports no variance across runs [arxiv:2012.05999 §IV-B]. ## When to use which The review gives no rule that links a constraint to a family. The rows below pair each paper's stated scope with a constraint that scope fits. | Situation | Use | Why | |---|---|---| | The model is a Lasso and you solve it over a grid of penalties | Safe screening (EDPP) | It is safe: it never drops a feature the solution needs, and it runs before any solver. The reported path times are 33× to 231× shorter on three data sets [arxiv:1211.3966 tab. 1]. It assumes the previous penalty's solution is exact [arxiv:1211.3966 §2.2]. | | You already train a linear model and want fewer inputs from the same fit | Embedded, L1 penalty | One fit gives the weights and the zeros. The paper reports 18 rule weights at zero, with TPR 99.35% for L1 against 99.34% for L2 [arxiv:2406.13547 §4.4] [arxiv:2406.13547 tab. 1]. | | Very few samples, and you want a readable list of candidate features | Embedded, LASSO with CV-chosen penalty, treated as exploratory | The stylometry study fits one LASSO per domain and reads the non-zero coefficients as features [arxiv:2407.03646 §2.2]. With ten texts the selected set can change with the fold split, and the penalty and folds are not stated [arxiv:2407.03646 p. 6]. | | The feature count is large, and an exhaustive subset search is out of reach | Greedy wrapper (e.g. sequential forward selection) | The subset count is 2^n [arxiv:2301.08826 §2.2]. The review names the greedy method but does not measure its cost or accuracy [arxiv:2301.08826 §2.2.2]. | | Many features, a search budget you can afford, and you can tune the search yourself | Population-based wrapper (e.g. PSO) | The swarm survey finds PSO highest on mean accuracy for SVM and AdaBoost, and the mean wrapper run time is 38.4 s for PSO against 45.36 s for ACO [arxiv:2008.04103 tab. 6]. It states no population size, iteration count or fitness function, and each family is represented by one method [arxiv:2008.04103 §4] [arxiv:2008.04103 §4.3]. | | Many candidates, and you want a cheap, inspectable ranking before training a small model | Univariate ranking, then top-k (e.g. top 1, 3 or 30) | Each feature is scored alone, with no subset search. The authors report AUCs of 80–90% for the top 30 on 10 medications [arxiv:2605.04666 tab. 2], and they note the greedy step may miss the best set [arxiv:2605.04666 p. 5]. | | Cheap screening before any model is trained | Filter (e.g. chi-square, information gain) | The review names these tools only as used in cited studies [arxiv:2301.08826 §2.2.3]. It gives no speed or accuracy comparison. The swarm survey's filters score lower on average than its wrappers [arxiv:2008.04103 §4.4]. | | Imbalanced, high-dimensional data | Per-feature filter scores, with j chosen by you | The class-imbalance review calls feature selection essential here, but gives no value of j and no evidence for the claim [arxiv:1305.1707 §2.3] [arxiv:1305.1707 §5]. | For a real choice, read the original studies behind the reported results. They report on different datasets, so the table above does not rank the methods. The timing, TPR and AUC rows come from different papers and setups, and should not be read against each other. Whatever method you pick, run it inside each training fold. The medical-imaging card flags that its selection ran before cross-validation, which makes the reported accuracies likely optimistic [arxiv:2609.07408 alg. 3]. The clinical card says the split is under-specified and that ranking should use only the training part [arxiv:2605.04666 p. 3]. The swarm survey also selects on the training part only [arxiv:2008.04103 §4], but it scales each dataset before the split, which can leak test-set range into the features [arxiv:2008.04103 §4.1]. ## Implementation sketch The review names sequential forward selection [arxiv:2301.08826 §2.2.2] but gives no code. The loop below is the standard procedure, written here, not taken from the review. ```pseudocode # Sequential forward selection (wrapper), standard procedure # Input: features F (n of them), target size k, score(S) for a subset S # Output: selected subset S with k features def forward_select(F, k, score): # Start with no features selected S = [] while len(S) < k: # Try each unused feature with S and keep the best-scoring one best = max((f for f in F if f not in S), key=lambda f: score(S + [f])) S.append(best) return S ``` `score` is the caller's choice, for example cross-validated accuracy of a model. The review does not fix one. The univariate variant needs one score per feature and no loop over subsets: sort the features by their own score and keep the first k, as in [Valko and Hauskrecht, 2026](/papers/2605.04666). Run it on the training fold only. The embedded and screening variants need no such loop. L1 is one fit, and screening is one test per feature for each penalty. The EDPP loop is in the [Wang et al., 2012](/papers/1211.3966) card. The swarm survey's protocol is a repeated split, with the selection step on the training part and the accuracy on the test part [arxiv:2008.04103 §4]; the survey gives it as its evaluation loop, not as a selection method. ## Contradictions None between the papers' claims. The reported numbers are not comparable, as noted above. Within one paper, the counts differ: - [Scano et al., 2024](/papers/2406.13547) says 18 of 60 CRS rules can be dropped [arxiv:2406.13547 §1], while the method uses 52 rule features [arxiv:2406.13547 §3]. - [Khan, 2020](/papers/2012.05999) says 13 attributes are used [arxiv:2012.05999 §IV-B], its attribute table lists 14 [arxiv:2012.05999 tab. 3], and its UCI accuracy table lists 16 total features [arxiv:2012.05999 tab. 7]. - [Georgiou, 2024](/papers/2407.03646) finds primary stress has a large LASSO coefficient while its univariate test is not significant [arxiv:2407.03646 tab. 2] [arxiv:2407.03646 tab. 3]. The author calls it the most influential phonological predictor [arxiv:2407.03646 §4]. ## Papers - [Shaik et al., 2023](/papers/2301.08826): a review of NLP for student feedback. It names the three selection families and reports results from cited studies, but runs no experiments of its own. - [Longadge and Dongre, 2013](/papers/1305.1707): a review of class-imbalance remedies. It lists feature selection as one of three families, with a user-set number of features and no experiments of its own. - [Wang et al., 2012](/papers/1211.3966): safe screening rules (DPP and EDPP) that drop Lasso features before solving. The rules are safe and work with any solver; the speed-ups are author-reported. - [Scano et al., 2024](/papers/2406.13547): ModSec-Learn, which learns weights for SQL-injection firewall rules, and L1 sets some of them to zero. The results are author-reported on one balanced test set. - [Valko and Hauskrecht, 2026](/papers/2605.04666): ranks temporal EHR features by single-feature AUC and trains a linear SVM on the top 1, 3 or 30. The results are author-reported on one hospital's records. - [Bhat and Szczuko, 2026](/papers/2609.07408): RFE picks image statistics for nine classifiers on spiral drawings. The selection step is run before cross-validation, so the reported accuracies are likely optimistic. - [Rostami et al., 2020](/papers/2008.04103): a survey of eleven swarm-intelligence feature-selection families, with one wrapper and one filter run per family on six UCI datasets. The comparison is author-run, uses no significance test, and states no method settings. - [Khan, 2020](/papers/2012.05999): an IoT heart-disease classifier that picks attributes with a modified cuttlefish search and tunes its network with elephant-herd optimisation. The results are author-reported on the authors' datasets, and parts of the data are generated. - [Georgiou, 2024](/papers/2407.03646): a ten-essay stylometry study that fits one LASSO per linguistic domain. The features are exploratory, the sample is very small, and the study releases no data or code.