Johan Mazel

Unsupervised Network Intrusion Detection Systems: Detecting the Unknown without Knowledge

Traditional Network Intrusion Detection Systems (NIDSs) rely on either specialized signatures of previously seen attacks, or on expensive and difficult to produce labeled traffic datasets for user-profiling to hunt out network attacks. Despite being opposite in nature, both approaches share a common downside: they require the knowledge provided by an external agent, either in terms of signatures or as normal-operation profiles. In this paper we present UNIDS, an Unsupervised Network Intrusion Detection System capable of detecting unknown network attacks without using any kind of signatures, labeled traffic, or training. UNIDS uses a novel unsupervised outliers detection approach based on Sub-Space Clustering and Multiple Evidence Accumulation techniques to pin-point different kinds of network intrusions and attacks such as DoS/DDoS, probing attacks, propagation of worms, buffer overflows, illegal access to network resources, etc. We evaluate UNIDS in three different traffic datasets, including the well-known KDD99 dataset as well as real traffic traces from two operational networks. We particularly show the ability of UNIDS to detect unknown attacks, comparing its performance against traditional misuse-detection-based NIDSs. In addition, we also evidence the supremacy of our outliers detection approach with respect to different previously used unsupervised detection techniques. Finally, we show that the algorithms used by UNIDS are highly adapted for parallel computation, which permits to drastically reduce the overall analysis time of the system.

Hashdoop: A MapReduce framework for network anomaly detection

Anomaly detection is essential for preventing network outages and maintaining the network resources available. However, to cope with the increasing growth of Internet traffic, network anomaly detectors are only exposed to sampled traffic, so harmful traffic may avoid detector examination. In this paper, we investigate the benefits of recent distributed computing approaches for real-time analysis of non-sampled Internet traffic. Focusing on the MapReduce model, our study uncovers a fundamental difficulty in order to detect network traffic anomalies by using Hadoop. Since MapReduce requires the dataset to be divided into small splits and anomaly detectors compute statistics from spatial and temporal traffic structures, special care should be taken when splitting traffic. We propose Hashdoop, a MapReduce framework that splits traffic with a hash function to preserve traffic structures and, hence, profits of distributed computing infrastructures to detect network anomalies. The benefits of Hashdoop are evaluated with two anomaly detectors and fifteen traces of Internet backbone traffic captured between 2001 and 2013. Using a 6-node cluster Hashdoop increased the throughput of the slowest detector with a speed-up of 15; thus, enabling real-time detection for the largest analyzed traces. Hashdoop also improved the overall detectors accuracy as splits emphasized anomalies by reducing the surrounding traffic.

UNADA: unsupervised network anomaly detection using sub-space outliers ranking

Current network monitoring systems rely strongly on signature-based and supervised-learning-based detection methods to hunt out network attacks and anomalies. Despite being opposite in nature, both approaches share a common downside: they require the knowledge provided by an expert system, either in terms of anomaly signatures, or as normal-operation profiles. In a diametrically opposite perspective we introduce UNADA, an Unsupervised Network Anomaly Detection Algorithm for knowledge-independent detection of anomalous traffic. UNADA uses a novel clustering technique based on Sub-Space-Density clustering to identify clusters and outliers in multiple low-dimensional spaces. The evidence of traffic structure provided by these multiple clusterings is then combined to produce an abnormality ranking of traffic flows, using a correlation-distance-based approach. We evaluate the ability of UNADA to discover network attacks in real traffic without relying on signatures, learning, or labeled traffic. Additionally, we compare its performance against previous unsupervised detection methods using traffic from two different networks.

Steps Towards Autonomous Network Security: Unsupervised Detection of Network Attacks

The unsupervised detection of network attacks represents an extremely challenging goal. Current methods rely on either very specialized signatures of previously seen attacks, or on expensive and difficult to produce labeled traffic data-sets for profiling and training. In this paper we present a completely unsupervised approach to detect attacks, without relying on signatures, labeled traffic, or training. The method uses robust clustering techniques to detect anomalous traffic flows, sequentially captured in a temporal sliding-window basis. The structure of the anomaly identified by the clustering algorithms is used to automatically construct specific filtering rules that characterize its nature, providing easy-to-interpret information to the network operator. In addition, these rules are combined to create an anomaly signature, which can be directly exported towards standard security devices like IDSs, IPSs, and/or Firewalls. The clustering algorithms are highly adapted for parallel computation, which permits to perform the unsupervised detection and construction of signatures in an on-line basis. We evaluate the performance of this new approach to discover and to build signatures for different network attacks without any previous knowledge, using real traffic traces. Results show that knowledge-independent detection and characterization of network attacks is possible, opening the door to a whole new generation of autonomous security algorithms.

Knowledge-independent traffic monitoring: Unsupervised detection of network attacks

The philosophy of traffic monitoring for detection of network attacks is based on an acquired knowledge perspective: current techniques detect either the well-known attacks on which they are programmed to alert, or those anomalous events that deviate from a known normal operation profile or behavior. In this article we discuss the limitations of current knowledge-based strategy to detect network attacks in an increasingly complex and ever evolving Internet. In a diametrically opposite perspective, we place the emphasis on the development of unsupervised detection methods, capable of detecting network attacks in a changing environment without any previous knowledge of either the characteristics of the attack or the baseline traffic behavior. Based on the observation that a large fraction of network attacks are contained in a small fraction of traffic flows, we demonstrate how to combine simple clustering techniques to accurately identify and characterize malicious flows. To show the feasibility of such a knowledge-independent approach, we develop a robust multiclustering-based detection algorithm, and evaluate its ability to detect and characterize network attacks without any previous knowledge, using packet traces from two real operational networks.

A taxonomy of anomalies in backbone network traffic

The potential threat of network anomalies on Internet has led to a constant effort by the research community to design reliable detection methods. Detection is not enough, however, because network administrators need additional information on the nature of events occurring in a network. Several works try to classify detected events or establish a taxonomy of known events. But, these works are non-overlapping in terms of anomaly type coverage. On the one hand, existing classification methods use a limited set of labels. On the other hand, taxonomies often target a single type of anomaly or, when they have wider scope, fail to present the full spectrum of what really happens in the wild. We thus present a new taxonomy of network anomalies with wide coverage of existing work. We also provide a set of signatures that assign taxonomy labels to events. We present a preliminary study applying this taxonomy with six years of real network traffic from the MAWI repository. We classify previously documented anomalous events and draw to main conclusions. First, the taxonomy-based analysis provides new insights regarding events previous classified by heuristic rule labeling. For example, some RST events are now classified as network scan response and the majority of ICMP events are split into network scans and network scan responses. Moreover, some previously unknown events now account for a substantial number of all UDP network scans, network scan responses and port scans. Second, the number of unknown events decreases from 20 to 10% of all events with the proposed taxonomy as compared to the heuristic approach.

sub-space clustering and evidence accumulation for unsupervised network anomaly detection

Network anomaly detection has been a hot research topic for many years. Most detection systems proposed so far employ a supervised strategy to accomplish the task, using either signature-based detection methods or supervised-learning techniques. However, both approaches present major limitations: the former fails to detect unknown anomalies, the latter requires training and labeled traffic, which is difficult and expensive to produce. Such limitations impose a serious bottleneck to the development of novel and applicable methods in the near future network scenario, characterized by emerging applications and new variants of network attacks. This work introduces and evaluates an unsupervised approach to detect and characterize network anomalies, without relying on signatures, statistical training, or labeled traffic. Unsupervised detection is accomplished by means of robust data-clustering techniques, combining Sub-Space Clustering and multiple Evidence Accumulation algorithms to blindly identify anomalous traffic flows. Unsupervised characterization is achieved by exploring inter-flows structure from multiple outlooks, building filtering rules to describe a detected anomaly. Detection and characterization performance of the unsupervised approach is extensively evaluated with real traffic from two different data-sets: the public MAWI traffic repository, and the METROSEC project data-set. Obtained results show the viability of unsupervised network anomaly detection and characterization, an ambitious goal so far unmet.

HIDDEN: Hausdorff Distance Based Intrusion Detection Approach DEdicated to Networks

DoS attacks represent a big threat for the Internet. While most of attack detection techniques are based on passive monitoring of traffic, we propose a detection method, HIDDEN, based on active measurements, the objective being to make possible the real-time detection and classification of DoS attacks, without intrusive probing. The originality of our contribution relies on the use of the entropy function computed from probabilities of time series of measured ICMP request/echo delays. However, the evaluation of the method exhibits a dramatic number of false positives. It has then been enriched by the use of the Hausdorff distance on probabilities of time series, which significantly decreases the number of false positives. In addition, a method for discriminating ICMP attacks from others (TCP/UDP attacks) using icmp_seq has been added. Experiments for evaluating the effectiveness of the approach have been run on the French operational RENATER network, on which artificial attacks have been generated using TFN2K [14]. Results exhibit that TCP, UDP and ICMP DoS attacks have been accurately detected in less than 1 second.

0day anomaly detection made possible thanks to machine learning

This paper proposes new cognitive algorithms and mechanisms for detecting 0day attacks targeting the Internet and its communication performances and behavior. For this purpose, this work relies on the use of machine learning techniques able to issue autonomously traffic models and new attack signatures when new attacks are detected, characterized and classified as such. The ultimate goal deals with being able to instantaneously deploy new defense strategies when a new 0day attack is encountered, thanks to an autonomous cognitive system. The algorithms and mechanisms are validated through extensive experiments taking advantage of real traffic traces captured on the Renater network as well as on a WIDE transpacific link between Japan and the USA.

An empirical mixture model for large-scale RTT measurements

Monitoring delays in the Internet is essential to understand the network condition and ensure the good functioning of time-sensitive applications. Large-scale measurements of round-trip time (RTT) are promising data sources to gain better insights into Internet-wide delays. However, the lack of efficient methodology to model RTTs prevents researchers from leveraging the value of these datasets. In this work, we propose a log-normal mixture model to identify, characterize, and monitor spatial and temporal dynamics of RTTs. This data-driven approach provides a coarse grained view of numerous RTTs in the form of a graph, thus, it enables efficient and systematic analysis of Internet-wide measurements. Using this model, we analyze more than 13 years of RTTs from about 12 millions unique IP addresses in passively measured backbone traffic traces. We evaluate the proposed method by comparison with external data sets, and present examples where the proposed model highlights interesting delay fluctuations due to route changes or congestion. We also introduce an application based on the proposed model to identify hosts deviating from their typical RTTs fluctuations, and we envision various applications for this empirical model.