Evaluating Federated Learning for Intrusion Detection in Internet of Things: Review and Challenges

In plain words

Intrusion detection systems watch network traffic and flag attacks. A common design sends all device traffic to a central server, which raises privacy concerns. Federated learning avoids that: each device trains on its own traffic and only shares model updates, and a coordinator combines those updates into one shared model.

The question here is how well this works when the traffic is uneven. Real IoT devices see very different traffic. One device may only ever see normal activity while another is hit by several kinds of attack. The authors split an IoT attack dataset into three client setups: uneven, evenly balanced, and a middle ground. They train a simple classifier that labels each flow as normal or as a specific attack. They compare the standard way of combining updates (FedAvg) with a more recent one (Fed+).

The simple averaging method struggles with uneven data, and the newer method holds up better. The authors also list open problems: choosing which devices take part, the privacy cost of protections, attacks on training, and realistic datasets for this setting.

TL;DR

• Evaluates federated intrusion detection on the CIC-ToN-IoT flows, which are derived from ToN_IoT, using multinomial logistic regression §4.2 §4.3.
• Builds three client settings: basic (one destination IP per client), balanced (equal class counts per client) and mixed (low-entropy clients removed, the rest thinned to a target entropy) §4.2.
• Compares FedAvg with Fed+ over 300 rounds §5. Fed+ gives the higher average client accuracy in all three federated settings §5.4.
• Accuracy is the headline metric, but the authors warn it is misleading on imbalanced clients §5.1. All numbers are author-reported single results, with no variance reported.
• Section 6 lists open challenges: device and client selection, lifecycle drift, poisoning, communication, privacy cost and the aggregator as a bottleneck §6.

Problem

Centralised IoT intrusion detection sends device traffic to a data centre. This raises privacy concerns, latency concerns and bandwidth overhead on constrained networks §2. Federated learning (FL) is proposed as the fix, but FL-based IDS for IoT was at an early stage, and many proposals used unrealistic settings or data distributions §2 §3.

Most public IoT IDS datasets cannot be split by device or IP address, which FL needs. The authors pick ToN_IoT because it can be partitioned that way, and it has a better benign-to-attack ratio and more attack types than the alternatives compared in Table 2 §4.1 tab. 2.

Assumptions:

• Each FL client is one IP address; the authors split by destination IP §4.2.
• Classifier inputs are treated as independent, as multinomial logistic regression requires §4.3.
• Under FedAvg, skewed non-IID client data can degrade convergence §4.4.

Method

Overview. The paper uses a single model family and varies only the data distribution and the aggregation rule. Everything is run in a simulated federation. A centralised run is reported only as context and is not a fair comparison §5.

1. Source data. Start from CIC-ToN-IoT, derived from the ToN_IoT pcap files with CICFlowMeter, which yields 83 features. Drop non-numeric features such as flow ID. The paper does not list which features were dropped §4.2. Keep the 10 destination IP addresses with the most samples. This gives 4,404,084 samples, which is 82.29% of CIC-ToN-IoT §4.2 tab. 3.

2. Client imbalance measure. Normalised Shannon entropy of a client's class counts. The result is 0 for one class and 1 for perfectly equal classes §4.2.

   \text{Entropy} = \frac{-\sum_{i=1}^{k} \frac{c_i}{n} \log \frac{c_i}{n}}{\log k} §4.2

   Here n is the client's number of flows, k the number of attack classes, and c_i the size of class i §4.2. The paper's equations are unnumbered in the source, so cite them by section.

3. Basic scenario. Each client holds the traffic of one IP address. This is the natural, highly skewed split §4.2.1 tab. 3.

4. Balanced scenario. Take a portion of the dataset and give each of the 10 clients the same number of samples per class. All clients then have the same entropy §4.2.2 tab. 3.

5. Mixed scenario.

   • Keep the clients whose entropy is above 0.2. These are clients 0, 2, 4 and 5 §4.2.3 tab. 3.
   • Remove samples from the predominant classes with a simple instance-selection step, until each client's entropy lies between 0.66 and 0.71 §4.2.3.
   • The paper does not say which samples are removed or how the stop is reached §4.2.3.

6. Classifier. Multinomial logistic regression (softmax regression). Class probabilities are a softmax over a per-class linear score, and the loss is cross-entropy §4.3. The paper's softmax and loss are unnumbered; cite by section.

7. Federated loop. Each round the coordinator sends the global weights to the clients, each client trains locally on its own data, and the coordinator aggregates the returned weights §2. The evaluation runs 300 rounds, with one local epoch per round §5.

   PseudocodeCopy

   # Federated training with a pluggable aggregation rule  [§2] [§5]
   # Input:  clients, each holding local flows and labels; 300 rounds
   # Output: global weights of the multinomial logistic regression
   
   def federated_train(clients, rounds, aggregate):
       # Initialisation is not stated in the paper  [§5]
       global_weights = init_weights()
       for _ in range(rounds):
           # Each client trains one epoch on its own data only  [§5]
           local_weights = []
           for client in clients:
               local_weights.append(train_one_epoch(global_weights, client))
           # Combine local updates with FedAvg or Fed+  [§4.4]
           global_weights = aggregate(global_weights, local_weights)
       return global_weights

8. FedAvg. Global weights are the average of the local weights, each weighted by the client's share of the total data size §4.4.

   w_i = \sum_k \frac{d_i}{D}\, w_i^k §4.4

   D is the total data size and d_i the size of party k's data §4.4. The paper states the objective as the mean of the local losses §4.4.

   PseudocodeCopy

   # FedAvg aggregation  [§4.4]
   # Input:  local_weights (one vector per client); sizes (data per client)
   # Output: new global weights
   
   def fedavg(global_weights, local_weights, sizes):
       # Total data size D across the selected clients  [§4.4]
       total = sum(sizes)
       # Weight each client by its share of the data  [§4.4]
       return sum(
           (size / total) * weights
           for size, weights in zip(sizes, local_weights)
       )

9. Fed+. Fed+ relaxes the requirement that all clients converge to one model. Its objective is the mean local loss plus α_i times a distance B between the model x and a central point C(X) of x §4.4. The paper gives no update rule, no choice of B or C, and no value for α_i. It refers to an external Fed+ paper (arXiv:2009.06303) for the details §4.4 [fed+paper]. That paper has no card here, so nothing from it is used.

   \text{min}\, F(x) = \frac{1}{D}\sum f_i(x) + \alpha_i B(x, C(X)) §4.4

10. Evaluation. Accuracy, precision, recall, F1 and false positive rate (FPR). The multiclass averages are micro, macro and weighted; the paper reports weighted averages because the scenarios are imbalanced §5.

Hyperparameters

Name	Value	Where	Notes
Clients (parties)	10	§4.2 tab. 3 config_agg.yml	One destination IP per client
Selected samples	4,404,084 (82.29% of CIC-ToN-IoT)	§4.2	Top 10 destination IPs by sample count
Features	83 extracted; non-numeric removed	§4.2	Final count and list not stated
Mixed-scenario clients	0, 2, 4, 5	§4.2.3 tab. 3	Only four clients appear in the mixed block of Table 3
Entropy threshold for mixed	> 0.2	§4.2.3
Entropy target for mixed	0.66 to 0.71	§4.2.3
Training rounds	300	§5 config_agg.yml
Local epochs per round	1	§5 config_agg.yml	Config sets max_iter: 1; reading it as one epoch is inferred
Model	multinomial logistic regression (softmax)	§4.3
Optimiser	SGD (scikit-learn SGDClassifier)	§5 config_party0.yml	Learning rate, schedule and batch size not stated
Loss	log loss (cross-entropy)	§5 §4.3
Regularisation	L2, strength not stated	§5
Input normalisation	yes; method not stated	§5
Train/test split	80/20	§5	Per client or global, and stratification, not stated
Initial weights	not stated
Aggregation weights	d_i / D	§4.4
Fed+ α_i, B, C	not stated	§4.4
Termination accuracy	0.9 in the config	config_agg.yml	Not in the paper; effect not described
Metric averaging	weighted	§5
Framework	IBM Federated Learning (IBMFL)	§1 §5	Version not stated in the paper
Hardware	not stated
Repeated runs or seeds	not stated

Results

Author-reported. Setup: CIC-ToN-IoT derived clients (10 total, 4 in mixed), multinomial logistic regression, 300 rounds, IBMFL, weighted metrics. Baseline is a distributed run where each client trains alone. A centralised run gives 0.724 accuracy, which the authors call not comparable and cite against 0.77 from the ToN_IoT paper §5.

Mean client accuracy, read from the text of §5.4 fig. 11:

Setting	Distributed (no FL)	FedAvg	Fed+
Basic	0.8718	starts 0.8725, drops slowly (end value not stated)	about 0.8725, constant
Balanced	0.9065	0.8349 start, 0.88 at about round 50, about 0.87 later	0.8569 start, rises to 0.9039, then stable
Mixed	0.877	0.8157 start, 0.8698 at about round 10, 0.8423 at the end	0.8498 start, 0.8876 at about round 50, 0.8869 at the end

Source: §5.1 §5.2 §5.3 §5.4 fig. 2–fig. 11. The authors state Fed+ gives the higher average accuracy for all federated settings §5.4.

Per-party accuracy in the basic setting, with Table 4 tab. 4:

Party	Distributed	Federated (Fed+)
0	0.5526	1.0
1	1	0.7293
2	0.9435	1.0
3	1	0.9402
4	0.7283	0.9434
5	0.6525	0.9525
6	0.9412	0.9513
7	1	0.5566
8	0.9493	1.0
9	0.9508	0.6527

Other reported values, all from the text:

• Basic, Fed+: parties 2 and 8 have precision and recall of 0 and FPR of 1, despite high accuracy §5.1 fig. 4.
• Mixed, Fed+: parties 2 and 4 have the best metrics, FPR 0.28, F1 0.91, recall 0.925 and precision 0.9 §5.3 fig. 10.

Paper vs code

Code: the authors' repository at commit 9a788e2 (full hash in frontmatter). It is linked in the paper's bibliography as the dataset source §4.2 and its owner matches an author. The repository holds only a README and 11 IBMFL YAML configs: one aggregator and ten parties. It does not contain training code, evaluation code, a Fed+ handler, or the data README.md config_agg.yml.

• Aggregation: the paper reports Fed+ results §5. The only aggregator config uses FedAvgFusionHandler. No Fed+ configuration was found in the repository config_agg.yml.
• Rounds and clients: 300 rounds and 10 parties match the paper config_agg.yml.
• Local work: max_iter: 1 is consistent with one epoch per round config_agg.yml.
• Termination: the config sets a termination accuracy of 0.9 config_agg.yml. The paper says nothing about early stopping, so its effect on the reported 300-round curves is unknown.
• Model: SklearnSGDFLModel from a pickled model file that is not in the repository. Loss, penalty, learning rate and regularisation strength cannot be checked config_party0.yml.
• Data: each party config points to a data_partyN.csv file on the author's machine. The handler is DDoSSklearnDataHandler, though the task is multiclass config_party0.yml. The scenario files are at a MEGA folder linked in the README README.md. Their licence was not checked here.
• Mixed and balanced configs: the repository has ten party configs only. It contains no separate configs for the balanced or mixed settings config_party0.yml.
• License: the repository has no LICENSE file, so no code is quoted here.

Pitfalls

• Accuracy is misleading on imbalanced clients. The authors say so themselves §5.1. A party with a single dominant class can score high accuracy with zero recall on its attack classes. In the basic setting, parties 2 and 8 show this §5.1 fig. 4. Report per-class recall, precision and F1 alongside accuracy.
• Text and table disagree on the basic scenario. The text says parties 1 and 3 hold only two XSS samples §4.2.1. Table 3 shows party 1 with 763,516 benign and 2 XSS samples, and party 3 with 519,804 benign and 2 XSS tab. 3. Use the table.
• Entropy definition is incomplete. The paper does not say whether k is all attack classes or only those present in a client §4.2. The log base is not stated, but it cancels in the normalisation. Table 3 entropies were not recomputed here, so the k choice is unverified.
• Balanced row is one row for ten clients. The row lists "0-9" with 43,549 samples. The per-class counts in that row sum to 43,549, so each client is read as having 43,549 samples. That reading is inferred tab. 3.
• Balanced and mixed construction is under-specified. The paper does not say which portion of the data is used for the balanced setting, or how the samples are chosen. It notes that clients in the balanced setting may hold other clients' samples, which has privacy consequences §4.2.2. The mixed removal procedure is also not given §4.2.3.
• Client participation per round is unclear. The text does not say whether all ten clients train in every round, or whether a selection happens. Table 1 lists "4/10" for this work tab. 1, which may refer to the four mixed-setting clients rather than a per-round sample.
• Fed+ cannot be implemented from this paper. The update rule, B, C and α_i are missing §4.4. The repository has no Fed+ code. Implement Fed+ from its own paper, not from this card.
• Evaluation protocol is not fully stated. The paper does not say whether a client's accuracy is the global model on that client's test split, or on a shared test set §5. It also does not say whether the 80/20 split is per client or global.
• Single runs, no variance. Repeated runs and seeds are not reported. Curve-level claims such as "stable after about 50 rounds" are read from figures and approximated by the authors §5 fig. 3.
• Recomputed mean checks out. The basic-setting Table 4 federated column averages to 0.8726 by hand, against 0.8725 in the text §5.4 tab. 4. The distributed column averages to 0.8718, which matches the text.
• Centralised comparison is not like-for-like. The authors exclude the centralised run as unfair, but still quote its 0.724 accuracy §5.
• Equations and figures are numbered by source order. Equations are unnumbered in the LaTeX. Table and figure numbers were counted in source order, and were not checked against the compiled PDF.
• Dataset licence. The scenario data derive from CIC-ToN-IoT. This card does not check the dataset licence and does not redistribute the data.

Lineage

• Builds on: FedAvg (McMahan et al., 2017) §2 §4.4. Fed+ (arXiv:2009.06303) is the aggregation the paper adds §4.4. The card does not link it, since no card exists for it. IBM Federated Learning (IBMFL) is the framework §1.
• Data: CIC-ToN-IoT, derived from ToN_IoT §4.2. The ToN_IoT paper is cited for its 0.77 accuracy §5.
• Compared against: a distributed, one-client-per-model baseline §5. Related FL-IDS works are classified in Table 1 tab. 1 §3.
• Superseded by: not known from content in this wiki.

Related

ConceptFederated learningClients train on their own data and a server combines what they send, so raw data stays on devices. What a client sends sets the trade-off: weights (FedMA), class embeddings (FedProto) or distilled predictions (FedDF, FedNTD). FedDyn corrects each client's objective, FedGen trains a data-free generator, Ditto keeps a personal model per device, FairFed reweights for fairness, k-FED clusters once.ConceptFederated aggregationHow a federated server turns client updates into one global model. FedAvg weights clients by data share. FedNTD changes the local loss. Fed+ adds a per-client penalty without its update rule. FairFed reweights clients by fairness gap. FedDF distils client logits on unlabeled data. FedGen trains a server generator for data-free distillation. FEDL also averages gradients.Paper · 2022-11-23Vertical Federated Learning: Concepts, Advances and ChallengesSurvey of vertical federated learning, where parties hold different features of the same users. Covers the training protocol, communication and effectiveness methods, privacy attacks and defenses, fairness, applications and open challenges. Proposes VFLow, a meta-objective that treats VFL design as constrained multi-objective optimisation. No new experiments; no official code.Paper · 2022-03-22Federated Class-Incremental LearningGLFC targets federated class-incremental learning: clients learn new classes over time with a 2,000-sample exemplar memory. Each client uses a gradient-compensated loss and a relation distillation loss; a proxy server rebuilds prototype samples from shared gradients and keeps the best old global model. Author-reported: 4.4% to 15.1% higher average accuracy than adapted baselines.Paper · 2021-11-08Federated Learning Based on Dynamic RegularizationFedDyn: each active device minimises its local loss plus a per-round dynamic penalty (a linear term from its stored gradient and a quadratic pull toward the server model), so device optima align with global stationary points. The server averages active models and corrects them with a running state h. Author-reported: up to 9.4x fewer models sent than SCAFFOLD on CIFAR-10.Paper · 2021-10-04Trustworthy AI: From Principles to PracticesSurvey and practitioner guide. Defines eight aspects of AI trustworthiness (robustness, generalization, explainability, transparency, reproducibility, fairness, privacy, accountability), maps existing approaches onto the AI lifecycle, and proposes TrustAIOps, a continuous MLOps-style workflow. No new method or experiments in the main text; no numeric results.

Source and license

Adapted from “Evaluating Federated Learning for Intrusion Detection in Internet of Things: Review and Challenges” by Enrique Mármol Campos, Pablo Fernández Saura, Aurora González-Vidal, José L. Hernández-Ramos, Jorge Bernal Bernabe, Gianmarco Baldini, Antonio Skarmeta (arXiv:2108.00974), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/Enrique-Marmol/Evaluating-FL-for-Intrusion-Detection-in-IoT-review-and-challenges at commit 9a788e205508 (none (no LICENSE file in repository)). Any quoted code is shown under that license.