Feature selection

Overview

Feature selection keeps some of the candidate features and drops the rest. The review places it after feature extraction, which turns text into vectors such as bag-of-words, TF-IDF or word embeddings 2301.08826 §2.1. The invariant it states is the size of the search space: n features give 2^n possible subsets 2301.08826 §2.2.

Nine cards on this wiki list the concept, each from a different setting. The review gives the three families but runs no experiments 2301.08826 §5. A safe-screening paper gives exact rules that drop Lasso features before the Lasso is solved 1211.3966 §2.3.3. A web-firewall paper uses L1 regularisation to set some rule weights to zero, which selects rules 2406.13547 §4.4. A clinical paper ranks temporal features one at a time by AUC and trains a linear SVM on the top few 2605.04666 p. 3. A medical-imaging paper selects image statistics with recursive feature elimination (RFE) 2609.07408 p. 10. A class-imbalance review lists feature selection as one of three families of remedies 1305.1707 §2.3. A swarm-intelligence survey runs wrapper and filter methods on six UCI datasets 2008.04103 §4. A heart-disease paper uses a metaheuristic search to pick attributes 2012.05999 §II-B. A stylometry study keeps the features that a LASSO model gives non-zero weight 2407.03646 §2.2.

Variants

Filter methods

Filter methods score features without training the final model. The review names the family 2301.08826 §2.2 but gives no procedure for it. Chi-square and information gain appear only as the tools used in cited studies 2301.08826 §2.2.3. Card: Shaik et al., 2023.

The class-imbalance review suggests filters for high-dimensional data: each feature is scored on its own, and the user picks the number j of features to keep 1305.1707 §2.3. It gives no value for j and no criterion beyond the per-feature scores. Card: Longadge and Dongre, 2013.

The clinical paper ranks each feature by its single-feature AUC for one prediction target, with no joint search 2605.04666 p. 3. It then keeps the top 1, 3 or 30 features 2605.04666 tab. 1 2605.04666 tab. 2. The ranking is cheap because it scores each feature alone. The authors say that with more than 9,000 candidates this greedy step may not find the best set 2605.04666 p. 5. Card: Valko and Hauskrecht, 2026.

The swarm survey runs four filter methods, one per study it chose to represent the filter side, and reports their mean accuracy over six datasets. On SVM the best filter averages 79.52%, from an ant-colony method, and the wrapper average for the same classifier is higher 2008.04103 §4.4. The survey also states that filters usually treat features as independent and tend to reach lower-quality subsets 2008.04103 §2.1. Card: Rostami et al., 2020.

Wrapper methods

Wrapper methods search over subsets and score each one with a model. The review names sequential forward selection as an example 2301.08826 §2.2.2. It adds one feature at a time, so it visits far fewer subsets than the 2^n total. Card: Shaik et al., 2023.

The medical-imaging paper uses RFE to rank the eleven image statistics, separately for each model, and keeps the features ranked 2 or better [arxiv:2609.07408 alg. 3] 2609.07408 p. 10. The paper does not state the estimator, step size or fold count used by RFE 2609.07408 alg. 3. The feature sets it uses in the experiments were then chosen by reading correlation matrices, with no stated rule 2609.07408 p. 13 2609.07408 p. 22. Card: Bhat and Szczuko, 2026.

Population-based (swarm) search is another way to explore subsets. The swarm survey evaluates one wrapper per swarm family, eleven in all, and reports for each the mean over six UCI datasets with three classifiers [arxiv:2008.04103 §3] 2008.04103 §4.3. On SVM and AdaBoost, particle swarm optimisation (PSO) has the highest mean. On Naive Bayes, the cuckoo method is first and PSO second, a gap of 0.16 points 2008.04103 §4.4. The authors say PSO does better than the other wrappers in most cases, but they run no significance test 2008.04103 §4.4. Its wrappers kept 7.21% of Arrhythmia's features and 0.58% of Colon's 2008.04103 tab. 5. The survey states no population size, iteration count, stopping rule or fitness function for any method 2008.04103 §4. Card: Rostami et al., 2020.

The heart-disease paper uses a modified cuttlefish search to pick attributes before its classifier is tuned 2012.05999 §II-B. It selects 7 of the 13 UCI attributes, while the compared methods select between 6 and 10, or keep all 13 2012.05999 tab. 6. The fitness function of the search and its population size are not stated 2012.05999 §II-E. Card: Khan, 2020.

Embedded methods

Embedded methods select features as part of training. The review gives L1 regularisation as an example and reports random-forest importance from a cited study 2301.08826 §2.2.3. An L1 penalty can set some weights to exactly zero, and the features with zero weight are then dropped.

ModSec-Learn trains linear SVMs, logistic regression (LR) and random forests on binary features, one per SQL-injection rule of the OWASP Core Rule Set, which either fired or did not 2406.13547 §3. At paranoia level 4, LR with L1 gives weight 0 to 18 rules, while the L2 model and ModSecurity use all of them 2406.13547 §4.4. LR with L1 reaches 99.35% TPR at 1% FPR there, against 99.34% for LR with L2 2406.13547 tab. 1. These are the authors' numbers on their balanced test set. The 18-rule count has not been checked against the released model. Card: Scano et al., 2024.

The stylometry study fits one binomial LASSO per linguistic domain, with the AI-or-human label as outcome, and takes the features with non-zero coefficients at the penalty that minimises cross-validated error 2407.03646 §2.2. Across the four domains it keeps 13 features, including primary stress, alveolar consonants and pronouns 2407.03646 tab. 3. The features are not scaled before fitting, so the coefficient sizes cannot be compared 2407.03646 §2.2. The study has ten texts in all, and the card notes that the penalty, folds and seed are not stated, so a rerun on another fold split may keep different features 2407.03646 p. 6. Card: Georgiou, 2024.

Safe screening for the Lasso

Safe screening is not a search over models. Before the Lasso is solved, it drops features whose coefficients are guaranteed to be zero, so the smaller problem gives the same answer as the full one 1211.3966 §1. The enhanced rule (EDPP) tests each feature against a small region around the dual solution at a nearby penalty 1211.3966 §2.3.3. Along a decreasing grid of penalties, each test uses the solution at the previous penalty [arxiv:1211.3966 §2.2]. Any Lasso solver can run after the screen 1211.3966 §1. The rules are derived for one scaling of the Lasso objective, so other conventions must be mapped first 1211.3966 §2.1.

The authors report these solver times for a 100-penalty path with the SLEP solver. On Prostate the solver alone takes 121.41 s and EDPP plus solver takes 3.70 s. On PIE the figures are 629.94 s and 4.13 s, and on MNIST 2566.26 s and 11.12 s 1211.3966 tab. 1. The paper also extends the rules to the group Lasso 1211.3966 §3. Card: Wang et al., 2012.

Reported results

The review quotes these accuracies from cited studies 2301.08826 §2.2.3. Each uses its own dataset and baselines, so they are not comparable.

The cards on this wiki report these results, each on its own data and setup. The clinical paper's linear SVM uses the top-k features from univariate AUC ranking 2605.04666 tab. 1 2605.04666 tab. 2. Its numbers are AUCs for one hospital's records, with no external baseline.

The Bhat and Szczuko card records no run with all eleven statistics, so its numbers do not isolate the effect of selection. Its RFE was also run before cross-validation, which the card flags as optimistic 2609.07408 alg. 3. The Khan card records no run that uses all 13 attributes with the same classifier, so its 98.2% cannot be credited to the selection step. The paper also names no split, and reports no variance across runs 2012.05999 §IV-B.

When to use which

The review gives no rule that links a constraint to a family. The rows below pair each paper's stated scope with a constraint that scope fits.

For a real choice, read the original studies behind the reported results. They report on different datasets, so the table above does not rank the methods. The timing, TPR and AUC rows come from different papers and setups, and should not be read against each other.

Whatever method you pick, run it inside each training fold. The medical-imaging card flags that its selection ran before cross-validation, which makes the reported accuracies likely optimistic 2609.07408 alg. 3. The clinical card says the split is under-specified and that ranking should use only the training part 2605.04666 p. 3. The swarm survey also selects on the training part only 2008.04103 §4, but it scales each dataset before the split, which can leak test-set range into the features 2008.04103 §4.1.

Implementation sketch

The review names sequential forward selection 2301.08826 §2.2.2 but gives no code. The loop below is the standard procedure, written here, not taken from the review.

# Sequential forward selection (wrapper), standard procedure
# Input:  features F (n of them), target size k, score(S) for a subset S
# Output: selected subset S with k features

def forward_select(F, k, score):
    # Start with no features selected
    S = []
    while len(S) < k:
        # Try each unused feature with S and keep the best-scoring one
        best = max((f for f in F if f not in S),
                   key=lambda f: score(S + [f]))
        S.append(best)
    return S

score is the caller's choice, for example cross-validated accuracy of a model. The review does not fix one.

The univariate variant needs one score per feature and no loop over subsets: sort the features by their own score and keep the first k, as in Valko and Hauskrecht, 2026. Run it on the training fold only.

The embedded and screening variants need no such loop. L1 is one fit, and screening is one test per feature for each penalty. The EDPP loop is in the Wang et al., 2012 card. The swarm survey's protocol is a repeated split, with the selection step on the training part and the accuracy on the test part 2008.04103 §4; the survey gives it as its evaluation loop, not as a selection method.

Contradictions

None between the papers' claims. The reported numbers are not comparable, as noted above. Within one paper, the counts differ:

• Scano et al., 2024 says 18 of 60 CRS rules can be dropped 2406.13547 §1, while the method uses 52 rule features [arxiv:2406.13547 §3].
• Khan, 2020 says 13 attributes are used [arxiv:2012.05999 §IV-B], its attribute table lists 14 2012.05999 tab. 3, and its UCI accuracy table lists 16 total features 2012.05999 tab. 7.
• Georgiou, 2024 finds primary stress has a large LASSO coefficient while its univariate test is not significant [arxiv:2407.03646 tab. 2] 2407.03646 tab. 3. The author calls it the most influential phonological predictor 2407.03646 §4.

Papers

• Shaik et al., 2023: a review of NLP for student feedback. It names the three selection families and reports results from cited studies, but runs no experiments of its own.
• Longadge and Dongre, 2013: a review of class-imbalance remedies. It lists feature selection as one of three families, with a user-set number of features and no experiments of its own.
• Wang et al., 2012: safe screening rules (DPP and EDPP) that drop Lasso features before solving. The rules are safe and work with any solver; the speed-ups are author-reported.
• Scano et al., 2024: ModSec-Learn, which learns weights for SQL-injection firewall rules, and L1 sets some of them to zero. The results are author-reported on one balanced test set.
• Valko and Hauskrecht, 2026: ranks temporal EHR features by single-feature AUC and trains a linear SVM on the top 1, 3 or 30. The results are author-reported on one hospital's records.
• Bhat and Szczuko, 2026: RFE picks image statistics for nine classifiers on spiral drawings. The selection step is run before cross-validation, so the reported accuracies are likely optimistic.
• Rostami et al., 2020: a survey of eleven swarm-intelligence feature-selection families, with one wrapper and one filter run per family on six UCI datasets. The comparison is author-run, uses no significance test, and states no method settings.
• Khan, 2020: an IoT heart-disease classifier that picks attributes with a modified cuttlefish search and tunes its network with elephant-herd optimisation. The results are author-reported on the authors' datasets, and parts of the data are generated.
• Georgiou, 2024: a ten-essay stylometry study that fits one LASSO per linguistic domain. The features are exploratory, the sample is very small, and the study releases no data or code.