Nour Moustafa

UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set)

One of the major research challenges in this field is the unavailability of a comprehensive network based data set which can reflect modern network traffic scenarios, vast varieties of low footprint intrusions and depth structured information about the network traffic. Evaluating network intrusion detection systems research efforts, KDD98, KDDCUP99 and NSLKDD benchmark data sets were generated a decade ago. However, numerous current studies showed that for the current network threat environment, these data sets do not inclusively reflect network traffic and modern low footprint attacks. Countering the unavailability of network benchmark data set challenges, this paper examines a UNSW-NB15 data set creation. This data set has a hybrid of the real modern normal and the contemporary synthesized attack activities of the network traffic. Existing and novel methods are utilised to generate the features of the UNSWNB15 data set. This data set is available for research purposes and can be accessed from the link.

The evaluation of Network Anomaly Detection Systems: Statistical analysis of the UNSW-NB15 data set and the comparison with the KDD99 data set

ABSTRACT Over the last three decades, Network Intrusion Detection Systems (NIDSs), particularly, Anomaly Detection Systems (ADSs), have become more significant in detecting novel attacks than Signature Detection Systems (SDSs). Evaluating NIDSs using the existing benchmark data sets of KDD99 and NSLKDD does not reflect satisfactory results, due to three major issues: (1) their lack of modern low footprint attack styles, (2) their lack of modern normal traffic scenarios, and (3) a different distribution of training and testing sets. To address these issues, the UNSW-NB15 data set has recently been generated. This data set has nine types of the modern attacks fashions and new patterns of normal traffic, and it contains 49 attributes that comprise the flow based between hosts and the network packets inspection to discriminate between the observations, either normal or abnormal. In this paper, we demonstrate the complexity of the UNSW-NB15 data set in three aspects. First, the statistical analysis of the observations and the attributes are explained. Second, the examination of feature correlations is provided. Third, five existing classifiers are used to evaluate the complexity in terms of accuracy and false alarm rates (FARs) and then, the results are compared with the KDD99 data set. The experimental results show that UNSW-NB15 is more complex than KDD99 and is considered as a new benchmark data set for evaluating NIDSs.

Novel Geometric Area Analysis Technique for Anomaly Detection using Trapezoidal Area Estimation on Large-Scale Networks

The prevalence of interconnected appliances and ubiquitous computing face serious threats from the hostile activities of network attackers. Conventional Intrusion Detection Systems (IDSs) are incapable of detecting these intrusive events as their outcomes reflect high false positive rates (FPRs). In this paper, we present a novel Geometric Area Analysis (GAA) technique based on Trapezoidal Area Estimation (TAE) for each observation computed from the parameters of the Beta Mixture Model (BMM) for features and the distances between observations. As this GAA-based detection depends on the methodology of anomaly-based detection (ADS), it constructs the areas of normal observations in a normal profile with those of the testing set estimated from the same parameters to recognise abnormal patterns. We also design a scalable framework for handling large-scale networks, and our GAA technique considers a decision engine module in this framework. The performance of our GAA technique is evaluated using the NSL-KDD and UNSW-NB15 datasets. To reduce the high-dimensional data of network connections, we apply the Principal Component Analysis (PCA) and evaluate its influence on the GAA technique. The empirical results show that our technique achieves a higher detection rate and lower FPR with a lower processing time than other competing methods.

Big Data Analytics for Intrusion Detection System: Statistical Decision-Making Using Finite Dirichlet Mixture Models

An intrusion detection system has become a vital mechanism to detect a wide variety of malicious activities in the cyber domain. However, this system still faces an important limitation when it comes to detecting zero-day attacks, concerning the reduction of relatively high false alarm rates. It is thus necessary to no longer consider the tasks of monitoring and analysing network data in isolation, but instead optimise their integration with decision-making methods for identifying anomalous events. This chapter presents a scalable framework for building an effective and lightweight anomaly detection system. This framework includes three modules of capturing and logging, pre-processing and a new statistical decision engine, called the Dirichlet mixture model based anomaly detection technique. The first module sniffs and collects network data while the second module analyses and filters these data to improve the performance of the decision engine. Finally, the decision engine is designed based on the Dirichlet mixture model with a lower-upper interquartile range as decision engine. The performance of this framework is evaluated on two well-known datasets, the NSL-KDD and UNSW-NB15. The empirical results showed that the statistical analysis of network data helps in choosing the best model which correctly fits the network data. Additionally, the Dirichlet mixture model based anomaly detection technique provides a higher detection rate and lower false alarm rate than other three compelling techniques. These techniques were built based on correlation and distance measures that cannot detect modern attacks which mimic normal activities, whereas the proposed technique was established using the Dirichlet mixture model and precise boundaries of interquartile range for finding small differences between legitimate and attack vectors, efficiently identifying these attacks.

A hybrid feature selection for network intrusion detection systems: Central points

Network intrusion detection systems are an active area of research to identify threats that face computer networks. Network packets comprise of high dimensions which require huge effort to be examined effectively. As these dimensions contain some irrelevant features, they cause a high False Alarm Rate (FAR). In this paper, we propose a hybrid method as a feature selection, based on the central points of attribute values and an Association Rule Mining algorithm to decrease the FAR. This algorithm is designed to be implemented in a short processing time, due to its dependency on the central points of feature values with partitioning data records into equal parts. This algorithm is applied on the UNSW-NB15 and the NSLKDD data sets to adopt the highest ranked features. Some existing techniques are used to measure the accuracy and FAR. The experimental results show the proposed model is able to improve the accuracy and decrease the FAR. Furthermore, its processing time is extremely short.

Identification of malicious activities in industrial internet of things based on deep learning models

Abstract Internet Industrial Control Systems (IICSs) that connect technological appliances and services with physical systems have become a new direction of research as they face different types of cyber-attacks that threaten their success in providing continuous services to organizations. Such threats cause firms to suffer financial and reputational losses and the stealing of important information. Although Network Intrusion Detection Systems (NIDSs) have been proposed to protect against them, they have the difficult task of collecting information for use in developing an intelligent NIDS which can proficiently detect existing and new attacks. In order to address this challenge, this paper proposes an anomaly detection technique for IICSs based on deep learning models that can learn and validate using information collected from TCP/IP packets. It includes a consecutive training process executed using a deep auto-encoder and deep feedforward neural network architecture which is evaluated using two well-known network datasets, namely, the NSL-KDD and UNSW-NB15. As the experimental results demonstrate that this technique can achieve a higher detection rate and lower false positive rate than eight recently developed techniques, it could be implemented in real IICS environments.

Flow Aggregator Module for Analysing Network Traffic

Network flow aggregation is a significant task for network analysis, which summarises the flows and improves the performance of intrusion detection systems (IDSs). Although there are some well-known flow analysis tools in the industry, such as NetFlow, sFlow and IPFIX, they can only aggregate one attribute at a time which increases networks’ overheads while running network analysis. In this paper, to address this challenge, we propose a new flow aggregator module which provides promising results compared with the existing tools using the UNSW-NB15 data set.

Anomaly Detection System Using Beta Mixture Models and Outlier Detection

An intrusion detection system (IDS) plays a significant role in recognising suspicious activities in hosts or networks, even though this system still has the challenge of producing high false positive rates with the degradation of its performance. This paper suggests a new beta mixture technique (BMM-ADS) using the principle of anomaly detection. This establishes a profile from the normal data and considers any deviation from this profile as an anomaly. The experimental outcomes show that the BMM-ADS technique provides a higher detection rate and lower false rate than three recent techniques on the UNSW-NB15 data set.

Towards Developing Network Forensic Mechanism for Botnet Activities in the IoT Based on Machine Learning Techniques

The IoT is a network of interconnected everyday objects called things that have been augmented with a small measure of computing capabilities. Lately, the IoT has been affected by a variety of different botnet activities. As botnets have been the cause of serious security risks and financial damage over the years, existing Network forensic techniques cannot identify and track current sophisticated methods of botnets. This is because commercial tools mainly depend on signature-based approaches that cannot discover new forms of botnet. In literature, several studies have conducted the use of Machine Learning ML techniques in order to train and validate a model for defining such attacks, but they still produce high false alarm rates with the challenge of investigating the tracks of botnets. This paper investigates the role of ML techniques for developing a Network forensic mechanism based on network flow identifiers that can track suspicious activities of botnets. The experimental results using the UNSW-NB15 dataset revealed that ML techniques with flow identifiers can effectively and efficiently detect botnets attacks and their tracks.

Probability Risk Identification Based Intrusion Detection System for SCADA Systems

As Supervisory Control and Data Acquisition (SCADA) systems control several critical infrastructures, they have connected to the internet. Consequently, SCADA systems face different sophisticated types of cyber adversaries. This paper suggests a Probability Risk Identification based Intrusion Detection System (PRI-IDS) technique based on analysing network traffic of Modbus TCP/IP for identifying replay attacks. It is acknowledged that Modbus TCP is usually vulnerable due to its unauthenticated and unencrypted nature. Our technique is evaluated using a simulation environment by configuring a testbed, which is a custom SCADA network that is cheap, accurate and scalable. The testbed is exploited when testing the IDS by sending individual packets from an attacker located on the same LAN as the Modbus master and slave. The experimental results demonstrated that the proposed technique can effectively and efficiently recognise replay attacks.