Mahbod Tavallaee

Toward developing a systematic approach to generate benchmark datasets for intrusion detection

In network intrusion detection, anomaly-based approaches in particular suffer from accurate evaluation, comparison, and deployment which originates from the scarcity of adequate datasets. Many such datasets are internal and cannot be shared due to privacy issues, others are heavily anonymized and do not reflect current trends, or they lack certain statistical characteristics. These deficiencies are primarily the reasons why a perfect dataset is yet to exist. Thus, researchers must resort to datasets that are often suboptimal. As network behaviors and patterns change and intrusions evolve, it has very much become necessary to move away from static and one-time datasets toward more dynamically generated datasets which not only reflect the traffic compositions and intrusions of that time, but are also modifiable, extensible, and reproducible. In this paper, a systematic approach to generate the required datasets is introduced to address this need. The underlying notion is based on the concept of profiles which contain detailed descriptions of intrusions and abstract distribution models for applications, protocols, or lower level network entities. Real traces are analyzed to create profiles for agents that generate real traffic for HTTP, SMTP, SSH, IMAP, POP3, and FTP. In this regard, a set of guidelines is established to outline valid datasets, which set the basis for generating profiles. These guidelines are vital for the effectiveness of the dataset in terms of realism, evaluation capabilities, total capture, completeness, and malicious activity. The profiles are then employed in an experiment to generate the desirable dataset in a testbed environment. Various multi-stage attacks scenarios were subsequently carried out to supply the anomalous portion of the dataset. The intent for this dataset is to assist various researchers in acquiring datasets of this kind for testing, evaluation, and comparison purposes, through sharing the generated datasets and profiles.

Automatic discovery of botnet communities on large-scale communication networks

Botnets are networks of compromised computers infected with malicious code that can be controlled remotely under a common command and control (C&C) channel. Recognized as one the most serious security threats on current Internet infrastructure, advanced botnets are hidden not only in existing well known network applications (e.g. IRC, HTTP, or Peer-to-Peer) but also in some unknown or novel (creative) applications, which makes the botnet detection a challenging problem. Most current attempts for detecting botnets are to examine traffic content for bot signatures on selected network links or by setting up honeypots. In this paper, we propose a new hierarchical framework to automatically discover botnets on a large-scale WiFi ISP network, in which we first classify the network traffic into different application communities by using payload signatures and a novel cross-association clustering algorithm, and then on each obtained application community, we analyze the temporal-frequent characteristics of flows that lead to the differentiation of malicious channels created by bots from normal traffic generated by human beings. We evaluate our approach with about 100 million flows collected over three consecutive days on a large-scale WiFi ISP network and results show the proposed approach successfully detects two types of botnet application flows (i.e. Blackenergy HTTP bot and Kaiten IRC bot) from about 100 million flows with a high detection rate and an acceptable low false alarm rate.

BotCop: An Online Botnet Traffic Classifier

A botnet is a network of compromised computers infected with malicious code that can be controlled remotely under a common command and control (C&C) channel. As one the most serious security threats to the Internet, a botnet cannot only be implemented with existing network applications (e.g. IRC, HTTP, or Peer-to-Peer) but also can be constructed by unknown or creative applications, thus making the botnet detection a challenging problem. In this paper, we propose a new online botnet traffic classification system, called BotCop, in which the network traffic are fully classified into different application communities by using payload signatures and a novel decision tree model, and then on each obtained application community, the temporal-frequent characteristic of flows is studied and analyzed to differentiate the malicious communication traffic created by bots from normal traffic generated by human beings. We evaluate our approach with about 30 million flows collected over one day on a large-scale WiFi ISP network and results show that the proposed approach successfully detects an IRC botnet from about 30 million flows with a high detection rate and a low false alarm rate.

Network Intrusion Detection and Prevention

With the complexity of todays networks, it is impossible to know you are actually secure. You can prepare your networks defenses, but what threats will be thrown at it, what combinations will be tried, and what directions they will come from are all unknown variables. Most medium and large-scale network infrastructures include multiple high-speed connections to the Internet and support many customer collaborative networks, thousands of internal users and various web servers. Many of these systems are faced with an ever-increasing likelihood of unplanned downtime due to various attacks and security breaches. In this environment of uncertainty, which is full of hackers and malicious threats, those systems that are the best at maintaining the continuity of their services (i.e., survive the attacks) enjoy a significant competitive advantage. Minimizing unexpected and unplanned downtime can be done by identifying, prioritizing and defending against misuse, attacks and vulnerabilities. Intrusion Detection and Prevention is a rapidly growing field that deals with detecting and responding to malicious network traffic and computer misuse. Intrusion detection is the process of identifying and (possibly) responding to malicious activities targeted at computing and network resources. Any hardware or software automation that monitors, detects or responds to events occurring in a network or on a host computer is considered relevant to the intrusion detection approach. Different intrusion detection systems provide varying functionalities and benefits. Network Intrusion Detection and Prevention: Concepts and Techniques provides detailed and concise information on different types of attacks, theoretical foundation of attack detection approaches, implementation, data collection, evaluation, and intrusion response. Additionally, it provides an overview of some of the commercially/publicly available intrusion detection and response systems. On the topic of intrusion detection system it is impossible to include everything there is to say on all subjects. However, we have tried to cover the most important and common ones. Network Intrusion Detection and Prevention: Concepts and Techniques is designed for researchers and practitioners in industry. This book is suitable for advanced-level students in computer science as a reference book as well.

A Novel Covariance Matrix Based Approach for Detecting Network Anomalies

During the last decade, anomaly detection has attracted the attention of many researchers to overcome the weakness of signature-based IDSs in detecting novel attacks. However, having a relatively high false alarm rate, anomaly detection has not been wildly used in real networks. In this paper, we have proposed a novel anomaly detection scheme using the correlation information contained in groups of network traffic samples. Our experimental results show promising detection rates while maintaining false positives at very low rates.

Online Classification of Network Flows

Online classification of network traffic is very challenging and still an issue to be solved due to the increase of new applications and traffic encryption. In this paper, we propose a hybrid mechanism for online classification of network traffic, in which we apply a signature-based method at the first level, and then we take advantage of a learning algorithm to classify the remaining unknown traffic using statistical features. Our evaluation with over 250 thousand flows collected over three consecutive hours on a large-scale ISP network shows promising results in detecting encrypted and tunneled applications compared to other existing methods.

Architecture and Implementation

Based on the place where data source are collected and analyzed, the IDS can be classified into centralized, distributed and agent based. In this Chapter, we discuss each category in terms of its architecture and implementation.

Alert Management and Correlation

Alert management includes functions to cluster, merge and correlate alerts. The clustering and merging functions recognize alerts that correspond to the same occurrence of an attack and create a new alert that merges data contained in these various alerts. The correlation function can relate different alerts to build a big picture of the attack. The correlated alerts can also be used for cooperative intrusion detection and tracing an attack to its source.

Hybrid Traffic Classification Approach Based on Decision Tree

Classifying network traffic is very challenging and is still an issue yet to be solved due to the increase of new applications and traffic encryption. In this paper, we propose a novel hybrid approach for the network flow classification, in which we first apply the payload signature based classifier to identify the flow applications and unknown flows are then identified by a decision tree based classifier in parallel. We evaluate our approach with over 100 million flows collected over three consecutive days on a large-scale WiFi ISP network and results show the proposed approach successfully classifies all the flows with an accuracy approaching 93%.

Automatic discovery of network applications: a hybrid approach

Automatic discovery of network applications is a very challenging task which has received a lot of attentions due to its importance in many areas such as network security, QoS provisioning, and network management In this paper, we propose an online hybrid mechanism for the classification of network flows, in which we employ a signature-based classifier in the first level, and then using the weighted unigram model we improve the performance of the system by labeling the unknown portion Our evaluation on two real networks shows between 5% and 9% performance improvement applying the genetic algorithm based scheme to find the appropriate weights for the unigram model.