Temporal Analysis of NetFlow Datasets for Network Intrusion Detection Systems

In plain words

A network intrusion detection system watches traffic and flags attacks. Many public datasets for training such systems store each conversation as one summary record, called a flow. Those records usually leave out timing: when a flow started and ended, and how far apart its packets arrived.

This paper rebuilds four well-known benchmark datasets in one common flow format that includes those timing details. It then looks at the data over time: how long flows last, how many flows of each attack type appear in a given minute, how traffic volume and the number of distinct addresses and ports change, and what a time-frequency picture of each attack type looks like.

The authors say many attacks leave visibly different patterns. They do not train a detector to test whether these patterns help one. The datasets are the main contribution. The analysis is exploratory.

TL;DR

• Four labelled NetFlow datasets (NF3-UNSW-NB15, NF3-BoT-IoT, NF3-ToN-IoT, NF3-CSE-CIC-IDS2018) with flow start and end times and inter-packet arrival time (IAT) statistics §4.1 tab. 1.
• Built by running the nProbe NetFlow exporter over the public PCAPs and matching flows to the ground-truth files by timestamp and 5-tuple §4.2.
• Results are descriptive only: class counts, flow-length histograms, per-minute series and spectrograms. No detector is trained or evaluated §5 §6.
• Use it as a data source, but check the labelling and the text-versus-table discrepancies listed under Pitfalls before relying on any single claim.

Problem

• Public NetFlow datasets for NIDS standardise the feature set, but they lack most temporal features. The authors argue this blocks sequential models and temporal analysis of attacks §1.
• The older benchmark datasets (KDD Cup 99, NSL-KDD) are described as outdated in their attack patterns §3.
• Feature sets differ across the datasets, which makes fair cross-dataset evaluation hard §3.
• Assumptions: the work relies on the original PCAP files and their ground-truth files being available for each benchmark §4.2.

Method

The paper has three parts: the dataset construction (§4), exploratory analysis (§5), and a closing discussion (§6). No new model or training procedure is proposed. There are no equations and no algorithm environments in the source.

1. Start from four benchmark captures. UNSW-NB15, BoT-IoT, ToN-IoT and CSE-CIC-IDS2018, each as public PCAP files with ground-truth event files §3 §4.2. The CSE-CIC-IDS2018 capture alone is described as more than 4,000 PCAP files totalling over 400 GB §4.2.

2. Export flows with nProbe. The authors run nProbe, a NetFlow exporter from Ntop, on each PCAP. They use NetFlow version 9, keep the original capture timestamps, and write plain-text output with # as the column separator §4.2. The nProbe command is quoted in the text §4.2.

3. Merge and label. The chronological text outputs are merged into CSV §4.2. Each flow gets a binary label (0 benign, 1 malicious) and a multi-class attack label, both matched from the ground-truth files using timestamps and the 5-tuple (source and destination IP, source and destination port, protocol) §4.2 fig. 1.

4. Feature set. The NF3 feature set equals the previous version's feature list plus temporal features §4.1 tab. 1. The temporal features fall in two groups:

• Flow timing: flow start and end time in milliseconds, stored as Unix timestamps §4.1.
• Inter-packet arrival time (IAT): minimum, maximum, average and standard deviation of the gaps between consecutive packets of a flow, computed for each direction (source to destination and destination to source) §4.1 tab. 1.

# Per-flow IAT statistics, one direction  [§4.1] [tab. 1]
# Input:  packet_times, arrival times in ms of one direction of one flow
# Output: minimum, maximum, average and standard deviation of the gaps

def iat_statistics(packet_times):
    # Gap between each pair of consecutive packets  [§4.1]
    gaps = consecutive_differences(packet_times)
    # Summary over all gaps of the flow  [§4.1]
    # Standard deviation type and single-packet flows: not stated
    return min(gaps), max(gaps), mean(gaps), std(gaps)

# Conversion and labelling  [§4.2] [fig. 1]
# Input:  pcap (raw capture), ground_truth (attack events)
# Output: labelled flow table, one row per flow

def build_labelled_flows(pcap, ground_truth):
    # Export flow records: NetFlow v9, original timestamps kept  [§4.2]
    text_records = run_nprobe(pcap, netflow_version=9, keep_timestamps=True)
    # Merge the chronological text files into one table  [§4.2]
    flows = merge_to_table(text_records)
    for flow in flows:
        # Binary (0 benign, 1 malicious) and attack-class labels  [§4.2]
        # Matching tolerance and unmatched flows: not stated
        flow.binary_label, flow.attack_class = match(flow, ground_truth)
    return flows

5. Analysis (exploratory, no model). Each analysis is shown on the four datasets, and most figures on one representative day per dataset.

• Flow length: histogram in 50 bins, flow lengths in seconds §5.1 fig. 2.
• IAT: distributions of the per-flow average IAT, shown per direction §5.2 fig. 3 fig. 4.
• Flows over time: flows per minute for each attack class on one representative day, volume on a log scale §5.3 fig. 5 tab. 4.
• Per-minute numerical series: IN_BYTES, IN_PKTS, OUT_BYTES and OUT_PKTS summed per minute §5.4.1 fig. 6.
• Per-minute categorical series: distinct source and destination IPs and ports per minute §5.4.2 fig. 7 tab. 5.
• Time-frequency: spectrograms of the most repeated pattern for each attack class in NF3-UNSW-NB15 §5.5 fig. 8.

# Per-minute series for one day  [§5.4] [fig. 6] [fig. 7]
# Input:  flows of one day, with timestamps and counters
# Output: per-minute volume totals and distinct-value counts

def minute_series(flows):
    # Group flows by minute; the timestamp used is not stated  [§5.4.1]
    for minute, group in group_by_minute(flows):
        # Sum IN_BYTES, IN_PKTS, OUT_BYTES, OUT_PKTS  [§5.4.1]
        volumes[minute] = sum_features(group, VOLUME_FEATURES)
        # Count distinct IPs and ports in the minute  [§5.4.2]
        uniques[minute] = count_distinct(group, CATEGORICAL_FIELDS)
    return volumes, uniques

6. Spectrograms. The authors apply a spectrogram, which they describe as a time-frequency representation of the traffic time series, to one pattern per attack class §5.5 fig. 8. The signal chosen, the window and the transform parameters are not stated §5.5.

Hyperparameters

Results

Author-reported. The paper reports dataset statistics and qualitative plots. It trains and evaluates no model, so there is no accuracy, F1 or baseline comparison. Setup: nProbe conversion on one Ubuntu 20.04 LTS machine §4.2. The baseline column is "none" throughout.

Class balance of the labelled datasets tab. 2:

Flows per attack class tab. 3:

I summed each column against the malicious and benign totals in tab. 2, and they match.

Attack types active on each day tab. 4. Active days: UNSW-NB15 3, CSE-CIC-IDS2018 11, ToN-IoT 10, BoT-IoT 6.

Distinct categorical values across the whole dataset tab. 5:

Qualitative claims (no metric): the abstract says "many attacks have unique patterns". The support is visual. For NF3-UNSW-NB15, the text says DoS and Worms share some similarity but remain distinct, and Fuzzers show a unique signature §5.5 fig. 8. No classifier, statistical test or effect size is reported for any of these claims.

Paper vs code

No official code. The paper links only the dataset page (https://staff.itee.uq.edu.au/marius/NIDS_datasets/, cited as uq_nids_datasets in §1 and §6). The searches I ran for official code:

• Paper and TeX source: no code link, only dataset and tool citations (nProbe by Ntop §4.2).
• Dataset page: lists CSV downloads and no code or scripts.
• GitHub search NF3-UNSW-NB15: one repository, owned by a user who is not an author. I did not use it.

The conversion depends on the third-party nProbe tool, not on code released with this paper §4.2.

Pitfalls

• Feature counts disagree. The text says the nProbe -T list has 57 features §4.2. The text also says the previous standard had 43 features §3. Table 1 has 45 non-bold rows (the earlier features) and 10 bold rows (the temporal features), 55 rows in total tab. 1. The paper does not reconcile these numbers. Take the exact column list from the released CSV headers, not from the table.
• Duplicate description. DURATION_OUT has the same description as DURATION_IN: "Client to Server stream duration (msec)" tab. 1. The direction of DURATION_OUT is therefore not clear from the paper.
• Timestamp meaning. Timestamps are Unix milliseconds, and they are the capture times from the PCAP. They are not the conversion time. This holds only with the original-timestamp option set §4.1 §4.2. Do not assume they equal ingest time.
• Export interval caps flow length. nProbe exports flows at intervals of at most 2 minutes by default §5.1. The paper does not discuss how this affects flow-length histograms, IAT statistics or labels. Inference from that setting: a single attack flow lasting longer than 2 minutes would appear as more than one record.
• Labelling is underspecified. Flows are matched to ground truth by timestamps and 5-tuple §4.2. The paper does not give the time tolerance, how ambiguous or unmatched flows are handled, or how a flow that spans a label boundary is labelled. Not stated.
• IAT edge cases. Single-packet directions, the standard deviation type (population or sample), and the units of IAT inputs are not stated §4.1 tab. 1.
• Minute binning rule. The paper does not say whether a flow is assigned to a minute by its start time, end time or both, nor how empty minutes are shown §5.3 §5.4.
• Text and table disagree on attack days. Table 4 shows BoT-IoT day 1 as Reconnaissance only, but the text says day 1 has "intense DDoS and DoS attacks" §5.3 tab. 4. The text says ToN-IoT day 5 has DDoS, DoS and Injection, but Table 4 lists Injection on day 6, not day 5 §5.3 tab. 4. The text says the CSE-CIC-IDS2018 day 5 shows DoS spikes, but Table 4 lists DDoS on day 5 §5.3 tab. 4. Check the labels against the data, not the prose.
• Figure captions and file names disagree. Captions for fig. 5, fig. 6 and fig. 7 say CSE-CIC-IDS2018 "Day 5" and ToN-IoT "Day 5". The source filenames read CICIDS2018Day1 and ToN_IoTDay3, so the captions and files do not agree on the day shown fig. 5 fig. 6 fig. 7.
• Unexplained interpretations. The text attributes intent to some patterns, for example possible data exfiltration, periodic signalling, or botnet command and control §5.1 §5.2 §5.4.1 §5.4.2. These are interpretations, not measured findings.
• Class imbalance. BoT-IoT is 99.7% malicious and UNSW-NB15 is 5.40% malicious tab. 2. Any model built on these datasets must handle that.
• Spectrogram details are missing. The window, hop, FFT size and normalisation are not stated. The signal chosen for each class is not stated. "The most repeated pattern" is not defined §5.5 fig. 8. Results are visual only.
• Overclaim risk. The abstract's "unique patterns" claim is a visual impression. It is not backed by a detection experiment in the paper, which lists ML evaluation as future work §6.
• Dataset licence is not stated in the paper. The paper's CC BY 4.0 licence covers only the paper. Do not redistribute the NF3 dataset files through this wiki. Link to the authors' page instead.
• Reproduction cost. The CSE-CIC-IDS2018 PCAP set is more than 400 GB in more than 4,000 files §4.2. Reconverting the datasets needs substantial storage.
• Version. The raw metadata records arXiv v3 (updated 2026-08-28) [raw/arxiv metadata]. The fetched LaTeX source is the version this card describes.

Lineage

• Builds on: the same group's earlier standard NetFlow datasets (NF1 and NF2), with the 43-feature set and the feature selection in SARHAN_SHAP. The sources are "Towards a standard feature set for network intrusion detection system datasets" (Mobile Networks and Applications) and "Evaluating Standard Feature Sets Towards Increased Generalisability and Explainability of ML-Based Network Intrusion Detection" (Big Data Research, 2022) §3 §4.
• Source datasets: UNSW-NB15, BoT-IoT, ToN-IoT and CSE-CIC-IDS2018 §3. Not compiled in this wiki.
• Compared against: no quantitative comparison. Earlier NetFlow datasets are described as lacking temporal features §1.
• Superseded by: not stated.