IGRF-RFE: A Hybrid Feature Selection Method for MLP-based Network Intrusion Detection on UNSW-NB15 Dataset

In plain words

Network intrusion detectors learn from records that describe each connection, and many of those columns are redundant or useless. Extra columns can make a learning model fit noise and perform worse, so picking the right subset of columns matters.

This paper proposes a two-step way to pick that subset. First, a quick screening keeps the columns that two different importance measures consider useful, and takes the union of the two lists. Second, the method removes columns one at a time from that shortlist, each time dropping the column whose removal leaves a small neural network most accurate on held-out data, and stops when several rounds bring no gain. The result is a smaller set of columns that the authors use to train the same neural network. On the UNSW-NB15 benchmark the smaller set does better than using all columns, and the authors argue it also needs fewer columns to do so.

TL;DR

• Two stages: an ensemble filter (information gain > 0.25 or random-forest importance > 0.02, union) p. 20 p. 21, then recursive feature elimination (RFE) with an MLP scored on validation accuracy alg. 2 p. 21.
• Final model: 2 hidden layers of 128 ReLU units, batch norm, softmax output, Adam, learning rate 0.0003, batch 64, up to 300 epochs, early stopping after 30 epochs p. 12 fig. 5.
• Author-reported: 23 features (20 numeric, 3 categorical) and 84.24% accuracy on a 6-class UNSW-NB15 test split, against 82.25% with all 42 features tab. 6 tab. 8.
• No official code was found [see Paper vs code].
• The paper has several internal inconsistencies in counts, equation numbers and the loss; see Pitfalls before implementing.

Problem

Redundant and irrelevant input features can cause overfitting and false positives in intrusion detection p. 2. Filter methods rank features independently and so miss interactions between them; wrapper methods capture interactions but train a model for every candidate subset and are expensive p. 5-6. The paper targets multi-class anomaly classification on UNSW-NB15, using the authors' 10% partition of the dataset p. 14.

Method

Data preparation (applied before the split, in the order of fig. 1 p. 6):

1. Drop the binary label column (keep attack_cat) and 44 rows with null values p. 15.
2. Remove four minority classes: Analysis, Backdoor, Shellcode and Worms p. 15.
3. Oversample the normal class by duplicating it ("double sampling") so that it reaches 49.5% of the training set p. 15-16 fig. 6.
4. One-hot encode the three categorical features, proto, service and state p. 16.
5. Min-max normalise each feature to [0, 1] p. 16-17 eq. 15.
6. Split the original test set, stratified by class, into a new validation set and a new test set; the original training set becomes the training set. The ratio is 68:16:16 p. 17-18 tab. 4.

Stage 1: ensemble filter on the 39 numeric features p. 8-9 fig. 3:

1. Remove duplicate rows from the training set only p. 19.
2. Compute information gain for each numeric feature (entropy in bits minus conditional entropy of the class given the feature) eq. 1 eq. 2 eq. 3 p. 6-7.
3. Compute random-forest importance from a forest of 1000 bootstrapped trees, averaging each feature's impurity decrease over the trees and normalising to [0, 1] alg. 1 p. 7-8 p. 20.
4. Keep features whose IG score is above 0.25 (22 features per the text) and whose RF score is above 0.02 (19 features per the text). Their union has 24 features, 17 of them in both lists fig. 10 p. 20-21.
5. Categorical features bypass this stage p. 8.

Stage 2: recursive feature elimination with an MLP alg. 2 p. 9-10:

1. Start with the 24 numeric candidates plus the 3 categorical features p. 21.
2. Score a feature set by training the MLP with 10 random seeds, 2022 to 2031, and averaging the validation accuracy p. 10 p. 21.
3. In each iteration, try removing each remaining numeric feature, keep the removal with the highest score, and record the feature subset whenever the score beats the best so far alg. 2.
4. Stop after 5 iterations without improvement (the paper's "patience" parameter, p = 5) p. 21.
5. Return the best subset. The paper reports 20 numeric plus 3 categorical features tab. 6.

# Stage 1a: remove duplicate rows from the training set only  [p. 19]
train = drop_duplicates(train)

# Stage 1b: information gain per numeric feature, scaled to [0, 1]  [eq. 1-3]
ig_scores = minmax(information_gain(train, numeric_features))

# Stage 1c: RF importance, 1000 trees, scaled to [0, 1]  [alg. 1] [p. 20]
rf_scores = minmax(rf_importance(train, numeric_features, trees=1000))

# Stage 1d: keep features above each threshold, then take the union  [p. 20]
keep_ig = [f for f in numeric_features if ig_scores[f] > 0.25]
keep_rf = [f for f in numeric_features if rf_scores[f] > 0.02]
candidates = sorted(set(keep_ig) | set(keep_rf))

# Stage 2: recursive feature elimination scored on validation accuracy  [alg. 2]
def score(features):
    # Mean validation accuracy over 10 seeds, 2022 to 2031  [p. 10] [p. 21]
    accuracies = []
    for seed in range(2022, 2032):
        model = train_mlp(train[features], train[label], seed)
        accuracies.append(accuracy(model, val[features], val[label]))
    return mean(accuracies)

# Categorical features are kept in every subset  [p. 9]
kept = list(candidates)
selected = kept + categorical_features
best_score = None        # initial value not stated  [alg. 2]
stalls = 0               # iterations without improvement  [p. 21]

while kept and stalls < 5:           # patience p = 5  [p. 21]
    # Try removing each remaining numeric feature  [alg. 2]
    trial = {}
    for f in kept:
        others = [g for g in kept if g != f]
        trial[f] = score(others + categorical_features)
    # Drop the feature whose removal scores best  [alg. 2]
    removed = max(trial, key=trial.get)
    kept.remove(removed)
    if best_score is None or trial[removed] > best_score:
        best_score = trial[removed]
        selected = kept + categorical_features
    else:
        stalls += 1
return selected

Classifier p. 11-12 fig. 5:

# MLP classifier  [p. 11-12]
def train_mlp(features, labels, seed):
    # Hidden layer 1: 128 ReLU units  [fig. 5]
    # Batch normalisation after the hidden layer  [p. 12] [eq. 9]
    # Hidden layer 2: 128 ReLU units, then batch normalisation  [fig. 5]
    # Softmax output, one unit per class  [p. 11] [eq. 7]
    # Adam, learning rate 0.0003, batch size 64, up to 300 epochs  [p. 12]
    # Stop after 30 epochs without validation-loss decrease,
    # then restore the best weights  [p. 12]
    ...

Key equations, exact as printed:

IG(Y, X) = H(Y) - H(Y \mid X) eq. 1

X_i = \frac{X_i - \mathrm{Mean}_i}{\mathrm{StdDev}_i} eq. 9

x'_i = \frac{x_i - \min(x_i)}{\max(x_i) - \min(x_i)} eq. 15

L(y, \hat{y}) = \frac{1}{m} \sum_{i=1}^{m} (y_i - \hat{y}_i)^2 eq. 8

Hyperparameters

Results

Author-reported. Setup: UNSW-NB15 10% partition, 6 classes after dropping the 4 minority classes, one MLP per feature set, single run, test set of 40,325 records tab. 4 p. 22. The paper reports no repeated runs or variance for the final model.

Final model, test set tab. 7 p. 22:

Accuracy 84.24% tab. 7. Per-class ROC AUC: DoS 0.95, Exploits 0.97, Fuzzers 0.89, Generic 1.00, Normal 0.99, Reconnaissance 0.97 fig. 12 p. 23. Largest confusions: 1806 of 2045 DoS samples predicted as Exploits, 889 Fuzzers predicted as Normal, 966 Normal predicted as Fuzzers fig. 11 p. 21-22.

Feature-set comparison, as printed in the table tab. 8 p. 23. The "Num." column disagrees with the text; see Pitfalls.

Comparison with other papers tab. 9 p. 24. The authors note that the numbers come from different data splits and that one 95.2% result (Prasad et al., ref. 27 of the paper) uses the full dataset p. 23.

Paper vs code

No official code. The paper has no code or project-page link; its only URL in the body is the dataset page p. 25. A GitHub repository search for "IGRF-RFE" returned one repository, Aryan-panda/IGRF-RFE-SHAP-IDS-UNSW-NB15 (MIT). Its owner is not an author of this paper, so it was not used. A search for "IGRF" returned unrelated repositories. Nothing was checked against code.

Pitfalls

• Feature counts disagree. The text gives 22 IG features, 19 RF features, 24 in the union and 17 in the intersection p. 20-21 fig. 10. Table 8 lists 25, 22, 27 and 20 for the same four sets tab. 8. The text's counts match Figure 10's Venn diagram, so use them as the check when you implement.
• Numeric-feature count. The text says 38 numeric features p. 14 and also 39 p. 8 p. 20. The column arithmetic supports 39: 42 features minus 3 categorical.
• Oversampling vs Table 4. Table 4 lists 56,000 normal training samples (32.9%) tab. 4 p. 15. The 49.5% figure p. 16 and the 5.4% and 8.0% shares for DoS and Fuzzers p. 21 imply the normal class was duplicated to about 112,000 (226,332 samples in total). The paper does not say which counts the final model trained on, whether validation was also oversampled, or whether "double sampling" means exact duplicates.
• Duplicate removal. The paper removes duplicates from the training set p. 19 but gives no count, and does not say whether this happened before or after the 68:16:16 split.
• Normalisation may leak. Min-max scaling is listed in the preprocessing order, before the split fig. 1 p. 6. The paper does not say whether the min and max come from the training data only.
• Loss. Equation 8 is mean squared error, yet the output is a softmax over classes eq. 7 eq. 8. The paper never names a classification loss. Check the code, if one is released, before choosing; treat eq. 8 as the printed definition only.
• Patience counter in Algorithm 2 runs backwards. When a removal improves the score the algorithm "increases patient counter"; otherwise it "reduces patient counter" alg. 2. The text says RFE stops when performance fails to improve over "cumulative five iterations" p. 21. This card follows the text: count non-improving iterations and stop at 5. The text's "cumulative" can also mean consecutive; the paper does not settle it.
• Algorithm 2 details. The initial best score is not stated; the card uses None. The candidate list temp_rm is appended to inside the loop without a reset alg. 2, so the removal set for each trial is ambiguous. The card removes exactly one feature per trial.
• Equation numbers in the text are wrong. The information-gain passages say "Eq. 7", "Eq. 8" and "Eq. 9" p. 7, but the printed equations are (1), (2) and (3). The card cites the printed numbers.
• Information gain on continuous features. The paper does not say how numeric features are binned, which estimator is used, or whether class entropy uses log base 2 only as printed in eq. 2 p. 7. Results depend on binning.
• Random-forest details missing. Split criterion (the text says Gini index or entropy p. 7), depth, leaf size, max features and random seed are not stated. Only "1000 trees" and bootstrap sampling are given.
• Batch normalisation. Eq. 9 has no learned scale or shift and no epsilon. Fig. 5 places batch norm after the ReLU of each hidden layer. Use the framework's standard layer and state the choice.
• One-hot counts are garbled. The text says the three categorical features contain "13, 9133 nominal values respectively" p. 16. Derive the category counts from the data.
• Text and table disagree on two numbers. Reconnaissance F1 is 78.83% in the text p. 22 and 0.7845 in Table 7. Exploits' FPR is 0.993 in the text and 0.0993 in Table 7. The table values are self-consistent; the text ones are not.
• RFE cost. Equation 13 counts pairwise evaluations only. Each evaluation trains the MLP 10 times, and each run can go to 300 epochs p. 10 p. 12. Algorithm 2 with 24 candidates has at most 24 + 23 + ... + 1 = 300 candidate evaluations, so about 3000 MLP trainings in the worst case. This is derived from Algorithm 2, not stated in the paper, and the paper gives no run times.
• Single run, no variance. The final accuracy is one run on one test split. The 2-point gain over all features is not tested against seed or split variance.
• Class-set dependence. Results are for 6 classes after dropping 4 minorities. They are not comparable with 10-class results, and the minority removal is a deliberate choice, not a general rule p. 15.
• Version. The PDF used here is the Journal of Big Data version (10:15, published 5 February 2023). Page citations use the printed "Page N of 26" numbers, because the section headings carry no numbers.

Lineage

• Builds on: hybrid filter-plus-wrapper selection, as in Hsu et al. (ref. 18 in the paper) p. 2; RFE, as in Kuhn and Johnson (ref. 38) p. 9; batch normalisation, as in Ioffe and Szegedy (ref. 40) p. 12.
• Dataset: UNSW-NB15, Moustafa and Slay (ref. 43) p. 14.
• Compared against: Kasongo and Sun (refs. 25, 49), Roy and Singh (ref. 48), Eunice et al. (ref. 26), Moustafa and Slay (ref. 44) tab. 9 p. 24.
• Superseded by: not known.