An Ensemble-based Machine Learning Framework for Advanced Distributed Denial of Service Attack Detection in Software Defined Networks

3.1. The Proposed Framework

The proposed framework leverages the centralized control plane paradigm inherent in SDN. The “Ryu” controller is employed in this study because of its Python-based modularity, adaptability for custom application development, and interoperability with various OpenFlow-enabled switches. The experimental network topology comprises multiple virtual hosts, a single OpenFlow-compliant switch, and the “Ryu” controller, which manages all flow rules. Specifically, one host functions as a web server hosting a basic HTTP application, while the remaining hosts generate both normal and malicious traffic. OpenFlow facilitates communication between the switch and the hosts. The switch transmits flow-level statistics to the “Ryu” controller, which, in turn, dictates forwarding policies, installs flow rules, and implements traffic shaping or blocking. This centralized control plane empowers network administrators to enforce granular security policies and perform real-time anomaly detection. Furthermore, the host with interface “h2-eth0” is designated as a monitoring point to observe traffic, analyze packets, and relay aggregated statistics to the classification module. This design aims to provide the “Ryu” controller with comprehensive network awareness, enabling it to apply immediate countermeasures upon detecting malicious traffic sources.

To accurately characterize the behavior of both normal and attack traffic, the system extracts 15 statistical features from observed flows. Following each capture period, the packet capture script gathers and normalizes a range of metrics for each active source IP (excluding the web server). These metrics are then transformed into a set of features that serve as the basis for traffic classification. Each feature is designed to act as a potential indicator for distinguishing between normal and malicious DDoS traffic.

The 15 attributes constitute a comprehensive set of measures intended to quantify both high-level and fine-grained characteristics of network traffic. Specifically, rate-based attributes (e.g., Request Rate, Packet Arrival Rate) are used to identify volumetric traffic surges, while connection-oriented metrics (e.g., TCP SYN Count, Failed Connection Attempts) are used to detect manipulations of the TCP handshake process commonly exploited in DDoS attacks. Furthermore, distribution-based attributes (e.g., Unique IP Count, Port Usage Distribution) provide insights into the scope of malicious host scanning activities. Each attribute’s mathematical derivation, practical significance, and specific utility in DDoS detection and mitigation are detailed in the subsequent subsections.

1. Request rate: The Request Rate measures the number of application-level requests (e.g., HTTP, FTP, or other protocol-specific requests) made by a source IP per unit time. Formally,

Where N_req is the total number of requests observed during the capture interval and Telapsed is the duration of the interval (in seconds). High request rates are generally a sign of application-layer DDoS (e.g., HTTP floods). By monitoring (R_req), it is straightforward to detect hosts generating more requests than normal.

2. Packet arrival rate: The packet arrival rate (R_pkt) quantifies how frequently a host transmits packets in seconds:

In volumetric DDoS attacks, attackers flood the network with an unusually large volume of packets. A sudden surge in (R_pkt) thus indicates beforehand that there can be a flood, marking normal bursts apart from more sustained malicious traffic.

3. Download rate: The download rate (R_bytes) is the average rate of data being transferred from a host over time:

Here, B_total is the quantity of bytes received by the host during the capture window. Bandwidth-depletion DDoS attacks attempt to exhaust a victim’s network link capacity by transferring large volumes of data. Tracking (R_bytes) helps identify this behavior.

4. Uptime: The uptime (U_time) measure indicates that the duration a host has been actively sending traffic during the period observed:

DDoS attacks in other instances are composed of high activity sustained over an extended period of time. Alternatively, legitimate clients will have more sporadic or short sessions. Thus, continually high Uptime can be an indicator of ongoing attack traffic as opposed to regular user-initiated behavior.

5. Flow duration variability: (V_flow) is the standard deviation of flow durations across hosts. If each flow i has duration Di, and D̅ is their mean, then:

Although there are legitimate long-lived and persistent flows (e.g., media streaming), the flows that result from DDoS attacks are normally high rates of extremely short or unusual flows. Therefore, analyzing (Vf low) reveals patterns indicating bursty or inconsistent traffic typical of malicious activity

6. Retransmission rate: The retransmission rate (R_retrans) is the amount of packets retransmitted by a host per second:

Misbehaving activities such as SYN floods or poorly misconfigured bots could generate a lot of duplicate packets or half-opened connections. An elevated (R_retrans) can signal malicious floods or repeated attempts at scanning for a victim server without completing up connections.

7. Unique IP count: The unique IP count (C_{Unique IP}) reveals how many distinct destination IP addresses a source host target:

In certain DDoS cases (or scanning attacks), an aggressive host rapidly probes many different IPs. However, a normal user tends to connect to fewer places in a short span of time, thus making an unusually high (CUnique IP) suspicious.

8. Sequential request patterns: Sequential Request Patterns (S seq) capture repeated requests to the same port from one host. Specifically,

Where Nreq(P) is the frequency of a host’s repeated requests for the same port PPP consecutively. Automated scripts usually use repeated instances of the same actions that repeatedly hammer the same service endpoint a classic indication of possible DDoS or brute-force attacks.

9. Flow count per host: The flow count per host (Nflows) is the sum of flows originated by a source:

Malicious hosts may establish dozens or hundreds of ephemeral flows in an attempt to overwhelm a server’s bandwidth. High rates of flows over a brief period are therefore an extremely strong indication of possible DDoS or network scanning.

10. Host communication frequency: Host Communication Frequency (fcomm) measures how many unique hosts a source communicates with per second:

This feature indicates whether a source is quickly propagating traffic to multiple targets (e.g., in a spread-based attack or scanning approach). In benign use cases, one typically sees smaller values that rise more slowly.

11. Port usage distribution: The Port Usage Distribution (Dports) gives the number of distinct destination ports utilized:

Attackers might attempt multi-port scanning or flooding to identify open services to attack. If one source is connecting to an unusually high number of different ports, it is either malicious probing or an advanced DDoS attack targeting multiple services.

12. TCP SYN count: The TCP SYN Count (Nsyn) measures how many TCP SYN packets a host sends.

Where δSYN (i) is 1 if the i -th packet is a SYN flag set, and 0 otherwise. SYN flood attacks are one of the most common types of DDoS, therefore large SYN counts within a short time interval a sure sign of potential flooding.

13. TCP ACK count: The TCP ACK Count (NACK) is the analogous measure for ACK packets:

Although the ACK flag alone is not always malicious, analyzing its relationship to SYN packets (e.g., the ratio of SYN to ACK) can highlight incomplete handshakes or unusual session behaviors that typify DDoS attempts.

14. Connections per second: Connections per second (Rconn) is the sum of new connections established by a host in one second:

Flooding attackers or active scanners have the tendency to establish many connections in quick succession to overload a target. Thus, a high (Rconn) distinguishes probable attackers from valid clients, which establish connections at a slower pace.

15. Failed connection attempts: Failed Connection Attempts (N_failed) refer to the number of connections attempted which fail to finish the TCP handshake. Monitoring can be done as follows:

Where NSYN is the number of SYN packets that were sent, and NSYN_ACKN is the number that was responded to with a SYN-ACK. In partial-handshake attacks (e.g., SYN floods), the attacker never finishes the server’s response, so there are many incomplete or failed sessions.

The selection of features for this study was guided by a literature review as well as domain knowledge pertaining to DDoS attack patterns. Request Rate, Packet Arrival Rate, TCP SYN Count, and Port Usage Distribution are a few of the features used extensively in previous studies on DDoS detection in SDN and traditional networks [2], [13], [21], [22]. But on top of that, we also have some sophisticated or innovative features such as Sequential Request Patterns and Failed Connection Attempts that exceed the typical ones in benchmark research. The reasoning behind these features is to attack application-layer session behavior and incomplete handshake patterns, especially relevant to SDN environments but commonly omitted in earlier research works. That combined approach provides more comprehensive and SDN-focused detection capacity.

3.2. Dataset Creation

The development of a labeled dataset is crucial for the effective training and evaluation of ML models in DDoS attack detection. To generate realistic traffic data, we implemented a two-phase approach over 6 days: An initial phase for normal traffic generation, followed by a phase dedicated to creating malicious traffic.

As shown in Fig. 3, the normal traffic phase spanned 2 days, during which typical user interactions with the web server were simulated. 11,186 aggregated samples which users sent normal traffic to the web server during these 2 days. These activities included browsing, page requests, occasional form submissions, and SSH interactions. Concurrently, background operations such as normal file downloads and MySQL queries were initiated to emulate a more diverse network environment.

Fig. 3. (Dataset creation) highlights the two-phase normal vs. malicious data gathering.

Upon capturing sufficient normal traffic data, the network configuration was transitioned to a malicious environment, which was sustained for 4 consecutive days. During this phase, various DDoS attack techniques were employed. A custom, multi-threaded Python script was utilized to generate continuous multi-threaded HTTP floods targeting the web server. In addition, high-volume SYN floods were launched using tools such as hping3 and iperf, alongside application layer resource exhaustion attacks. These attacks were designed to mimic common DDoS characteristics, 5 such as rapid request generation, incomplete TCP handshakes, and excessive consumption of server resources. In total, 11,469 aggregated samples in which bot machines participate in creating malicious traffic during the 4 days.

Throughout both phases, packet capture was consistently performed on the monitor host using Scapy. This ensured that all network activity originating from a source IP within a defined time window was recorded. Each captured network session was then labeled as either normal (0) or attack (1), resulting in a comprehensive dataset of benign and malicious network traffic.

The complete dataset used in this study is publicly available at https://www.kaggle.com/datasets/aramsaleem21/sddos-sdn-2025.

To clarify our choice of using a custom dataset instead of public alternatives like CICDDoS2019. A few of the engineered features used in this study are not computable or aggregable directly from the CICDDoS2019 dataset due to the granularity and content limitations of the available data. For example, Request Rate (req/sec) and Sequential Request Patterns depend on the application layer to recognize and count the distinct user interactions and their sequence. CICDDoS2019 does not have this application-level visibility.

Retransmission Rate depends on monitoring TCP sequence numbers to detect duplicate transmissions, which is not feasible without raw packet captures and sequence monitoring not available in CICDDoS2019. Failed Connection Attempts Rate must maintain connection state on a per-host basis, i.e., unmatched SYNs or aborted handshakes through RST/FIN flags. CICDDoS2019 does not maintain this level of TCP control information.

In contrast, our dataset was built from full packet captures of a controlled SDN setup and captures both application and transport-layer behavior for a diverse set of traffic scenarios. This provides the ability to extract more expressive, real-time, and SDN-specific traffic features and hence provides a more fertile ground for evaluating advanced machine learning (ML) models under conditions more closely aligned with real-world network operation.

Following the collection of a comprehensive dataset comprising DDoS and normal traffic, the subsequent step involved data preprocessing. This process transformed raw packet-level data into a structured dataset suitable for ML-based classification. The following subsections detail the preprocessing and validation procedures.

This preprocessing pipeline ensured the final dataset’s cleanliness, consistency, and appropriate scaling before classification.

3.3. Model Selection and Training Strategy

To comprehensively evaluate the effectiveness of ML techniques for DDoS detection, we selected eight classification models:

The preprocessed dataset was partitioned into training (70%), validation (15%), and test (15%) sets. This partitioning strategy ensures an unbiased evaluation of model performance on unseen data.

Within the training set, k-fold cross-validation was employed. In each iteration, a model was trained on k-1 folds, and performance metrics were computed on the remaining fold. This process was repeated k times, and the resulting average metrics mitigate variance associated with a single train-test split.

We optimized the hyperparameters of each model to maximize performance. For Gradient Boosting models, a grid search was conducted across the number of estimators, learning rate, maximum tree depth (max depth), and subsample ratio to identify optimal configurations. For AdaBoost and Bagging, we varied the base estimator depth and the number of estimators to achieve a balance between model complexity and overfitting. For the One-Class SVM, we experimented with RBF and linear kernels and tuned the “nu” parameter, which controls the fraction of outliers. Finally, for Bayesian Networks, we explored score-based and constraint-based structure learning algorithms and employed partial hill-climbing to identify effective network structures. The accuracy, area under curve (AUC), and F1-score metrics guided the selection of optimal hyperparameter configurations. The final model configurations were then retrained using the combined training and validation sets and evaluated on the held-out test set.

4.1. Local ML Detection

Each web server continuously monitors incoming traffic and analyzes features such as request rate, packet size, and unusual TCP flags. Upon detecting suspicious activity within a “five-second” window, the local ML model generates a list of the responsible IP addresses. This localized detection approach ensures rapid response times and avoids bottlenecks associated with centralized traffic analysis.

4.2. Secure IP List Transmission with CA-Based Sessions

Before transmitting the list of malicious IP addresses to the SDN controller, the web server establishes a secure communication channel using CA-issued certificates. The session establishment process involves the server first validating the controller’s certificate (issued by a trusted CA). Subsequently, the server encrypts a randomly generated session key with the controller’s public key. The controller then decrypts this encrypted session key using its corresponding private key, thereby acquiring the session key required for symmetric encryption. Finally, the server encrypts the list of malicious IP addresses using AES-256 encryption with the established session key, ensuring confidentiality and integrity during transmission.

4.3. Immediate Network-Wide Blocking

Once the IP list is decrypted, the SDN controller enforces network-wide blocking by commanding the OpenFlow switches to discard all packets from the labeled IP addresses. The blocking occurs in a short time and thus prevents malicious hosts from further flooding or compromising network resources. Because each web server handles traffic classification independently, the network gains horizontal scalability: Adding more servers proportionally enhances the overall detection capability of the network, while the central controller remains focused on coordinating drop rules rather than performing high-volume traffic analysis.

Fig. 4 provides a more step-by-step illustration of the entire process, from when an attack is identified by a server to where the bad traffic is dropped in the network fabric.

Fig. 4. Server-centric real-time distributed denial of service detection with CA-based security and immediate drop rules installed by the Software Defined Networking controller in this design, real-time detection at the server.

In such an architecture, real-time detection at the server level prevents making the SDN controller a computational bottleneck. Whereas CA-based session establishment ensures mutual authentication and secure key exchange, AES256 encryption protects the IP lists against eavesdropping and tampering. With fast blocking of malicious traffic at the switch level, the entire network becomes resilient against DDoS disruption, illustrating how a distributed model of detection nicely complements SDN’s centralization of policy enforcement.

5.1. Overall Classification Outcomes

Table 1 provides a comparative overview of the performance of the eight ML models, evaluated on the SDN-based DDoS dataset. The performance of each model was assessed using the following metrics: accuracy, precision, recall, F1-score, AUC, Cohen’s Kappa, Matthews correlation coefficient (MCC), FP rate (FPR), false-negative rate (FNR), and cross-validated accuracy.

Table 1: Classification performance metrics across different models

5.2. Confusion Matrices: Minimal Misclassifications

To better visualize model performance, confusion matrices were generated for each of the eight classifiers as shown in Fig. 5. These matrices illustrate the distribution of true positives, FP, true negatives (TN), and false negatives (FN). The confusion matrices highlight that ensemble methods, particularly gradient boosting techniques, achieve a strong balance between high detection rates and low false alarm rates.

Fig. 5. Confusion matrices for each classifier showing the distribution of true positives, false positives, true negatives, and false.

5.3. Cross-Validation and Consistency

Cross-validation accuracies remain consistently above 0.998 for LightGBM, CatBoost, Bagging, and XGBoost. This consistency across different folds indicates that their performance is robust and not attributable to overfitting on a specific train-test split. In contrast, cross-validation accuracy for One-Class SVM drops to 0.9505, revealing higher performance variability and emphasizing the importance of meticulous threshold tuning in anomaly-based detection systems.

6.1. Ensemble Methods and Their Dominance

The results clearly demonstrate the superior performance of ensemble-based approaches, particularly LightGBM and CatBoost. Their near-perfect scores highlight two key strengths:

The iterative refinement of gradient boosting, combined with the feature-rich dataset used in this study, likely contributes to the observed 0.9999 AUC scores in many cases. These high AUC values suggest that the malicious and benign traffic classes are distinctly separated within this dataset.

In computational efficiency, ensemble models such as LightGBM and CatBoost had fast convergence during training and low-latency predictions, which is suitable for real-time SDN deployments where response time is critical.

6.2. Trade-Offs with Anomaly Detection (One-Class SVM)

While One-Class SVM worked very well at identifying almost all attacks (recall = 0.9998), the sharp increase in FP (FPR = 0.1077) reflects a serious deficiency of anomaly-based detectors. It suggests that, when legitimate traffic significantly deviates from the baseline model, the model will end up marking benign requests as malicious. In operational scenarios, high FPR can lead to resource overhead in investigating benign events.

6.3. Bayesian Networks and Interpretability

Interestingly, the Bayesian Network achieved perfect precision (1.0000) at the expense of having a fairly elevated FNR (0.0035). That is, while it never misclassified benign traffic as an attack, it occasionally missed real attacks. Bayesian Networks do offer interpretability in the sense that they are graphical models whose conditional dependencies among features can make it easy to understand. For advanced research that aims to discover the underlying causal relations for attack patterns, this feature could provide new analytical contributions beyond bare classification.

Existing research has also explored the use of gradient boosting models such as LightGBM and CatBoost for DDoS detection. The majority of research, however, evaluates these models on typical data such as CICDDoS2019 or UNSW-15 and does not consider SDN-special traffic or real-time deployment. In contrast to our method, which is aimed at real SDN deployments and integrates these models with an ongoing mitigation process, existing literature does not. Table 2 indicates a comparison between this proposal and existing works.

Table 2: Accuracy comparison of LightGBM and XGBoost across prior DDoS detection studies

6.4. Implications for Advanced SDN-Based Research

With the network-level perspective of an SDN controller, high-performance ensemble models can be combined to offer a scalable and precise method for handling evolving DDoS threats. The distributed detection model with each server running the final classifier locally and securely notifying the controller of malicious IPs ensures low latency and effective utilization of resources. For mission-critical or large-scale deployments (e.g., data centers, cloud infrastructures), an extremely low FPR (essentially 0.0000) is required to avoid disrupting rightful traffic. Such minimal FPR (very nearly 0.0001) achieved by LightGBM and CatBoost shows that both models are well positioned for highly available scenarios fulfilling strict service-level agreements.

6.5. Limitations and Future Work

Although the ensemble methods have produced excellent results for this dataset, some of the potential avenues remain available for future work: