Ali A. Ghorbani

Toward developing a systematic approach to generate benchmark datasets for intrusion detection

In network intrusion detection, anomaly-based approaches in particular suffer from accurate evaluation, comparison, and deployment which originates from the scarcity of adequate datasets. Many such datasets are internal and cannot be shared due to privacy issues, others are heavily anonymized and do not reflect current trends, or they lack certain statistical characteristics. These deficiencies are primarily the reasons why a perfect dataset is yet to exist. Thus, researchers must resort to datasets that are often suboptimal. As network behaviors and patterns change and intrusions evolve, it has very much become necessary to move away from static and one-time datasets toward more dynamically generated datasets which not only reflect the traffic compositions and intrusions of that time, but are also modifiable, extensible, and reproducible. In this paper, a systematic approach to generate the required datasets is introduced to address this need. The underlying notion is based on the concept of profiles which contain detailed descriptions of intrusions and abstract distribution models for applications, protocols, or lower level network entities. Real traces are analyzed to create profiles for agents that generate real traffic for HTTP, SMTP, SSH, IMAP, POP3, and FTP. In this regard, a set of guidelines is established to outline valid datasets, which set the basis for generating profiles. These guidelines are vital for the effectiveness of the dataset in terms of realism, evaluation capabilities, total capture, completeness, and malicious activity. The profiles are then employed in an experiment to generate the desirable dataset in a testbed environment. Various multi-stage attacks scenarios were subsequently carried out to supply the anomalous portion of the dataset. The intent for this dataset is to assist various researchers in acquiring datasets of this kind for testing, evaluation, and comparison purposes, through sharing the generated datasets and profiles.

Y-means: a clustering method for intrusion detection

As the Internet spreads to each comer of the world, computers are exposed to miscellaneous intrusions from the World Wide Web. We need effective intrusion detection systems to protect our computers from these unauthorized or malicious actions. Traditional instance-based learning methods for intrusion detection can only detect known intrusions since these methods classify instances based on what they have learned. They rarely detect the intrusions that they have not learned before. In this paper, we present a clustering heuristic for intrusion detection, called Y-means. This proposed heuristic is based on the K-means algorithm and other related clustering algorithms. It overcomes two shortcomings of K-means: number of clusters dependency and degeneracy. The result of simulations run on the KDD-99 data set shows that Y-means is an effective method for partitioning large data space. A detection rate of 89.89% and a false alarm rate of 1.00% are achieved with Y-means.

A Survey of Visualization Systems for Network Security

Security Visualization is a very young term. It expresses the idea that common visualization techniques have been designed for use cases that are not supportive of security-related data, demanding novel techniques fine tuned for the purpose of thorough analysis. Significant amount of work has been published in this area, but little work has been done to study this emerging visualization discipline. We offer a comprehensive review of network security visualization and provide a taxonomy in the form of five use-case classes encompassing nearly all recent works in this area. We outline the incorporated visualization techniques and data sources and provide an informative table to display our findings. From the analysis of these systems, we examine issues and concerns regarding network security visualization and provide guidelines and directions for future researchers and visual system developers.

Network anomaly detection based on wavelet analysis

Signal processing techniques have been applied recently for analyzing and detecting network anomalies due to their potential to find novel or unknown intrusions. In this paper, we propose a new network signal modelling technique for detecting network anomalies, combining the wavelet approximation and system identification theory. In order to characterize network traffic behaviors, we present fifteen features and use them as the input signals in our system. We then evaluate our approach with the 1999 DARPA intrusion detection dataset and conduct a comprehensive analysis of the intrusions in the dataset. Evaluation results show that the approach achieves high-detection rates in terms of both attack instances and attack types. Furthermore, we conduct a full days evaluation in a real large-scale WiFi ISP network where five attack types are successfully detected from over 30 millions flows.

Botnet detection based on traffic behavior analysis and flow intervals

Botnets represent one of the most serious cybersecurity threats faced by organizations today. Botnets have been used as the main vector in carrying many cyber crimes reported in the recent news. While a significant amount of research has been accomplished on botnet analysis and detection, several challenges remain unaddressed, such as the ability to design detectors which can cope with new forms of botnets. In this paper, we propose a new approach to detect botnet activity based on traffic behavior analysis by classifying network traffic behavior using machine learning. Traffic behavior analysis methods do not depend on the packets payload, which means that they can work with encrypted network communication protocols. Network traffic information can usually be easily retrieved from various network devices without affecting significantly network performance or service availability. We study the feasibility of detecting botnet activity without having seen a complete network flow by classifying behavior based on time intervals. Using existing datasets, we show experimentally that it is possible to identify the presence of existing and unknown botnets activity with high accuracy even with very small time windows.

Network intrusion detection using an improved competitive learning neural network

The paper presents a novel approach for detecting network intrusions based on a competitive learning neural network. The performance of this approach is compared to that of the self-organizing map (SOM), which is a popular unsupervised training algorithm used in intrusion detection. While obtaining a similarly accurate detection rate as the SOM does, the proposed approach uses only one fourth of the computation time of the SOM. Furthermore, the clustering result of this method is independent of the number of the initial neurons. This approach also exhibits the ability to detect known and unknown network attacks. The experimental results obtained by applying this approach to the KDD-99 data set demonstrate that the proposed approach performs exceptionally in terms of both accuracy and computation time.

Detecting P2P botnets through network behavior analysis and machine learning

Botnets have become one of the major threats on the Internet for serving as a vector for carrying attacks against organizations and committing cybercrimes. They are used to generate spam, carry out DDOS attacks and click-fraud, and steal sensitive information. In this paper, we propose a new approach for characterizing and detecting botnets using network traffic behaviors. Our approach focuses on detecting the bots before they launch their attack. We focus in this paper on detecting P2P bots, which represent the newest and most challenging types of botnets currently available. We study the ability of five different commonly used machine learning techniques to meet online botnet detection requirements, namely adaptability, novelty detection, and early detection. The results of our experimental evaluation based on existing datasets show that it is possible to detect effectively botnets during the botnet Command-and- Control (C&C) phase and before they launch their attacks using traffic behaviors only. However, none of the studied techniques can address all the above requirements at once.

Alert correlation survey: framework and techniques

Managing raw alerts generated by various sensors are becoming of more significance to intrusion detection systems as more sensors with different capabilities are distributed spatially in the network. Alert Correlation addresses this issue by reducing, fusing and correlating raw alerts to provide a condensed, yet more meaningful view of the network from the intrusion standpoint. Techniques from a divers range of disciplines have been used by researchers for different aspects of correlation. This paper provides a survey of the state of the art in alert correlation techniques. Our main contribution is a two-fold classification of literature based on correlation framework and applied techniques. The previous works in each category have been described alongside with their strengths and weaknesses from our viewpoint.

Reputation Formalization for an Information–Sharing Multi–Agent System

We propose that through the formalization of concepts related to trust, a more accurate model of trust can be implemented. This paper presents a new model of trust that is based on the formalization of reputation. A multidisciplinary approach is taken to understanding the nature of trust and its relation to reputation. Through this approach, a practical definition of reputation is adopted from sociological contexts and a model of reputation is designed and presented.

Automatic discovery of botnet communities on large-scale communication networks

Botnets are networks of compromised computers infected with malicious code that can be controlled remotely under a common command and control (C&C) channel. Recognized as one the most serious security threats on current Internet infrastructure, advanced botnets are hidden not only in existing well known network applications (e.g. IRC, HTTP, or Peer-to-Peer) but also in some unknown or novel (creative) applications, which makes the botnet detection a challenging problem. Most current attempts for detecting botnets are to examine traffic content for bot signatures on selected network links or by setting up honeypots. In this paper, we propose a new hierarchical framework to automatically discover botnets on a large-scale WiFi ISP network, in which we first classify the network traffic into different application communities by using payload signatures and a novel cross-association clustering algorithm, and then on each obtained application community, we analyze the temporal-frequent characteristics of flows that lead to the differentiation of malicious channels created by bots from normal traffic generated by human beings. We evaluate our approach with about 100 million flows collected over three consecutive days on a large-scale WiFi ISP network and results show the proposed approach successfully detects two types of botnet application flows (i.e. Blackenergy HTTP bot and Kaiten IRC bot) from about 100 million flows with a high detection rate and an acceptable low false alarm rate.