Jungsuk Song

Toward a more practical unsupervised anomaly detection system

During the last decade, various machine learning and data mining techniques have been applied to Intrusion Detection Systems (IDSs) which have played an important role in defending critical computer systems and networks from cyber attacks. Unsupervised anomaly detection techniques have received a particularly great amount of attention because they enable construction of intrusion detection models without using labeled training data (i.e., with instances preclassified as being or not being an attack) in an automated manner and offer intrinsic ability to detect unknown attacks; i.e., 0-day attacks. Despite the advantages, it is still not easy to deploy them into a real network environment because they require several parameters during their building process, and thus IDS operators and managers suffer from tuning and optimizing the required parameters based on changes of their network characteristics. In this paper, we propose a new anomaly detection method by which we can automatically tune and optimize the values of parameters without predefining them. We evaluated the proposed method over real traffic data obtained from Kyoto University honeypots. The experimental results show that the performance of the proposed method is superior to that of the previous one.

A Generalized Feature Extraction Scheme to Detect 0-Day Attacks via IDS Alerts

Intrusion detection system (IDS) has played an important role as a device to defend our networks from cyber attacks. However, since it still suffers from detecting an unknown attack, i.e., 0-day attack, the ultimate challenge in intrusion detection field is how we can exactly identify such an attack. Unlike the existing approaches that investigate raw traffic data, we introduced a feature extraction method in order to detect such an attack from IDS alerts [J. Song et al., 2007]. However, there is a problem that it can be only applied to limited IDS products. In this paper, we present a generalized version of the feature extraction method. To this end, we define new 7 features using only the basic 6 features of IDS alerts; detection time, source address and port, destination address and port, and signature name. In order to detect 0-day attack from IDS alerts with new 7 features, we apply an unsupervised learning technique, One-class SVM, to them. We evaluated our method over the log data of IDS that is deployed in Kyoto University, and our experimental results show that it has capability to detect not only a type of 0-day attack detected in our previous study, but also several different types of 0-day attack.

Cooperation of Intelligent Honeypots to Detect Unknown Malicious Codes

Honeypot is one of the most popular tools to decoy attackers into our network, and to capture lots of information about the activity of malicious attackers. By tracing and analyzing collected traffic data, we can find out unknown malicious codes under an experimental stage before some codes become hazardous to an application. Although many honeypots have been proposed, there is a common problem that they can be detected easily by malicious attackers. This is very important in success or failure of honeypots because if once an attacker notices that he/she is working on a honeypot, we can no longer observe his/her malicious activities. In this paper, we propose two types of honeypot to collect unforeseen exploit codes automatically while maintaining their concealment against malicious attackers; cooperation based active honeypot and self-protection type honeypot. We have evaluated the proposed honeypots which are deployed in Kyoto University, and showed that they have capability to collect some unknown malicious codes.

A comprehensive approach to detect unknown attacks via intrusion detection alerts

Intrusion detection system(IDS) has played an important role as a device to defend our networks from cyber attacks. However, since it still suffers from detecting an unknown attack, i.e., 0-day attack, the ultimate challenge in intrusion detection field is how we can exactly identify such an attack. This paper presents a novel approach that is quite different from the traditional detection models based on raw traffic data. The proposed method can extract unknown activities from IDS alerts by applying data mining technique.We evaluated our method over the log data of IDS that is deployed in Kyoto University, and our experimental results show that it can extract unknown(or under development) attacks from IDS alerts by assigning a score to them that reflects how anomalous they are, and visualizing the scored alerts.

nicter: a large-scale network incident analysis system: case studies for understanding threat landscape

We have been developing the Network Incident analysis Center for Tactical Emergency Response (nicter), whose objective is to detect and identify propagating malwares. The nicter mainly monitors darknet, a set of unused IP addresses, to observe global trends of network threats, while it captures and analyzes malware executables. By correlating the network threats with analysis results of malware, the nicter identifies the root causes (malwares) of the detected network threats. Through a long-term operation of the nicter for more than five years, we have achieved some key findings that would help us to understand the intentions of attackers and the comprehensive threat landscape of the Internet. With a focus on a well-knwon malware, i. e., W32.Downadup, this paper provides some practical case studies with considerations and consequently we could obtain a threat landscape that more than 60% of attacking hosts observed in our dark-net could be infected by W32.Downadup. As an evaluation, we confirmed that the result of the correlation analysis was correct in a rate of 86.18%.

A Clustering Method for Improving Performance of Anomaly-Based Intrusion Detection System

Intrusion detection system (IDS) has played a central role as an appliance to effectively defend our crucial computer systems or networks against attackers on the Internet. The most widely deployed and commercially available methods for intrusion detection employ signature-based detection. However, they cannot detect unknown intrusions intrinsically which are not matched to the signatures, and their methods consume huge amounts of cost and time to acquire the signatures. In order to cope with the problems, many researchers have proposed various kinds of methods that are based on unsupervised learning techniques. Although they enable one to construct intrusion detection model with low cost and effort, and have capability to detect unforeseen attacks, they still have mainly two problems in intrusion detection: a low detection rate and a high false positive rate. In this paper, we present a new clustering method to improve the detection rate while maintaining a low false positive rate. We evaluated our method using KDD Cup 1999 data set. Evaluation results show that superiority of our approach to other existing algorithms reported in the literature.

Correlation Analysis between Spamming Botnets and Malware Infected Hosts

Many of recent cyber attacks are being launched by botnets for the purpose of carrying out large-scale cyber attacks such as spam emails, Distributed Denial of Service (DDoS), network scanning and so on. In many cases, these botnets consist of a lot of bots or zombie PCs which have been infected by a specific malware, and they try to propagate themselves into other victim systems through the Internet. In order to mitigate heavy damage of botnet based cyber attacks, it is needed to better understand the basic infrastructure of botnets as well as the underlying malwares of them. In this paper, we carried out correlation analysis between 10 spamming botnets identified by analyzing 3 weeks of spam emails in our previous work and malware infected hosts that observed at our darknets and honey pots. By comparing members (i.e., bots) of 10 spamming botnets with source hosts of dark net and honey pot traffic, we found that 7.2% ~ 37.5% of spamming botnets has been infected by four different malwares at least.

A robust feature normalization scheme and an optimized clustering method for anomaly-based intrusion detection system

Intrusion detection system(IDS) has played a central role as an appliance to effectively defend our crucial computer systems or networks against attackers on the Internet. Traditional IDSs employ signature-based methods or anomaly-based methods which rely on labeled training data. However, they have several problems, for example, it consumes huge amounts of cost and time to acquire the labeled training data, and they often experienced difficulty in detecting new types of attack. In order to cope with the problems, many researchers have proposed various kinds of algorithms for several years. Although they do not require labeled data for training and have the capability to detect unforeseen attacks, they are based on the assumption that the ratio of attack to normal is extremely small. However, the assumption may not be satisfied in a realistic situation because some attacks, most notably the denial-of-service attacks, consist of a large number of simultaneous connections. Consequently if the assumption fails, the performance of the algorithm will deteriorate. In this paper, we present a new normalization and clustering method that can overcome a limitation on the attack ratio of the training data. We evaluated our method using KDD Cup 1999 data set. Evaluation results show that performance of our approach is constant irrespective of an increase in the attack ratio.

A model of analyzing cyber threats trend and tracing potential attackers based on darknet traffic

In general, attackers carry out scanning or probing against a certain network when they start to attack their victims. Because of this, darknet is very useful to observe the scanning activities of attackers who want to find their victims that have security vulnerabilities in operating systems, applications, services, and so on. Thus, by observing and analyzing darknet traffic, it is able to obtain an insight into malicious activities that are happening on the Internet and to identify potential attackers who sent attack packets to the darknet. However, darknet has a fatal limitation that most of the darknet traffic has no payload data. This means that we cannot collect the real attack codes from the original darknet traffic. To cope with this problem, we propose a security monitoring and response model to analyze cyber threats trend and to trace potential attackers based on darknet traffic. We have evaluated the proposed model using one /24 darknet IP addresses and TMS alerts that were obtained from TMS. The experimental results provided the statistical information of all the incoming darknet traffic so that we could obtain the global cyber threats trend. Furthermore, the experimental results demonstrated that we could obtain malicious attack patterns and attack codes that were not detected by TMS. Copyright

A Methodology for Multipurpose DNS Sinkhole Analyzing Double Bounce Emails

DNS sinkhole is one of the powerful techniques to mitigate attack activities of bots, i.e., zombie PCs, by blocking the communication between C&C server and them. If a zombie PC sends a DNS query to our DNS server for communicating with its C&C server, our DNS server that contains domain blacklist of C&C servers returns IP address of our sinkhole server. As a result, since the zombie PC tries to communicate with our sinkhole server, it is unable to communicate with its C&C server. On the other hand, there are many cyber attacks caused by malicious URLs included in spam emails. Therefore, if we extract malicious URLs from spam emails and apply them into DNS sinkhole system, many of spam based attacks can be blocked. In this paper, we propose a methodology to enhance the capability of DNS sinkhole system by analyzing spam emails. Especially, we use double bounce emails, which do not have any valid sender and recipient addresses, as spam emails and extract malicious URLs from them. Our preliminary experimental results demonstrate that the existing domain blacklist of DNS sinkhole system is not effective. Thus, we design a new method collecting the malicious URLs from double bounce emails and show how new domain blacklist can be generated. With DNS sinkhole system using new domain blacklist, we will be able to early detect and block the latest malicious behaviors on the Internet.