Koji Nakao

Toward a more practical unsupervised anomaly detection system

During the last decade, various machine learning and data mining techniques have been applied to Intrusion Detection Systems (IDSs) which have played an important role in defending critical computer systems and networks from cyber attacks. Unsupervised anomaly detection techniques have received a particularly great amount of attention because they enable construction of intrusion detection models without using labeled training data (i.e., with instances preclassified as being or not being an attack) in an automated manner and offer intrinsic ability to detect unknown attacks; i.e., 0-day attacks. Despite the advantages, it is still not easy to deploy them into a real network environment because they require several parameters during their building process, and thus IDS operators and managers suffer from tuning and optimizing the required parameters based on changes of their network characteristics. In this paper, we propose a new anomaly detection method by which we can automatically tune and optimize the values of parameters without predefining them. We evaluated the proposed method over real traffic data obtained from Kyoto University honeypots. The experimental results show that the performance of the proposed method is superior to that of the previous one.

nicter: An Incident Analysis System Toward Binding Network Monitoring with Malware Analysis

We have been developing the Network Incident analysis Center for Tactical Emergency Response (nicter), whose present focus is on detecting and identifying propagating malwares such as worms, viruses, and bots. The nicter presently monitors darknet, a set of unused IP addresses, to observe macroscopic trends of network threats. Meantime, it keeps capturing and analyzing malware executables in the wild for their microscopic analysis. Finally, these macroscopic and microscopic analysis results are correlated in order to identify the root cause of the detected network threats. This paper describes a brief overview of the nicter, and possible contributions to the worldwide observatory of malicious behavior and attack tools (WOMBAT).

An incident analysis system NICTER and its analysis engines based on data mining techniques

Malwares are spread all over cyberspace and often lead to serious security incidents. To grasp the present trends of malware activities, there are a number of ongoing network monitoring projects that collect large amount of data such as network traffic and IDS logs. These data need to be analyzed in depth since they potentially contain critical symptoms, such as an outbreak of new malware, a stealthy activity of botnet and a new type of attack on unknown vulnerability, etc. We have been developing the Network Incident analysis Center for Tactical Emergency Response (NICTER), which monitors a wide range of networks in real-time. The NICTER deploys several analysis engines taking advantage of data mining techniques in order to analyze the monitored traffics. This paper describes a brief overview of the NICTER, and its data mining based analysis engines, such as Change Point Detector (CPD), Self-Organizing Map analyzer (SOM analyzer) and Incident Forecast engine (IF).

DAEDALUS-VIZ: novel real-time 3D visualization for darknet monitoring-based alert system

A darknet is a set of unused IP addresses whose monitoring is an effective way of detecting malicious activities on the Internet. We have developed an alert system called DAEDALUS (direct alert environment for darknet and livenet unified security), which is based on large-scale darknet monitoring. This paper presents a novel real-time 3D visualization engine called DAEDALUS-VIZ that enables operators to grasp visually and in real time a complete overview of alert circumstances and provides highly flexible and tangible interactivity. We describe some case studies and evaluate the performance of DAEDALUS-VIZ.

nicter: a large-scale network incident analysis system: case studies for understanding threat landscape

We have been developing the Network Incident analysis Center for Tactical Emergency Response (nicter), whose objective is to detect and identify propagating malwares. The nicter mainly monitors darknet, a set of unused IP addresses, to observe global trends of network threats, while it captures and analyzes malware executables. By correlating the network threats with analysis results of malware, the nicter identifies the root causes (malwares) of the detected network threats. Through a long-term operation of the nicter for more than five years, we have achieved some key findings that would help us to understand the intentions of attackers and the comprehensive threat landscape of the Internet. With a focus on a well-knwon malware, i. e., W32.Downadup, this paper provides some practical case studies with considerations and consequently we could obtain a threat landscape that more than 60% of attacking hosts observed in our dark-net could be infected by W32.Downadup. As an evaluation, we confirmed that the result of the correlation analysis was correct in a rate of 86.18%.

Behavior analysis of long-term cyber attacks in the darknet

Darknet monitoring provides us an effective way to countermeasure cyber attacks that pose a significant threat to network security and management. This paper aims to characterize the behavior of long term cyber attacks by mining the darknet traffic data collected by the nicter project. Machine learning techniques such as clustering, classification, function regression are applied to the study with promising results reported.

Correlation Analysis between Spamming Botnets and Malware Infected Hosts

Many of recent cyber attacks are being launched by botnets for the purpose of carrying out large-scale cyber attacks such as spam emails, Distributed Denial of Service (DDoS), network scanning and so on. In many cases, these botnets consist of a lot of bots or zombie PCs which have been infected by a specific malware, and they try to propagate themselves into other victim systems through the Internet. In order to mitigate heavy damage of botnet based cyber attacks, it is needed to better understand the basic infrastructure of botnets as well as the underlying malwares of them. In this paper, we carried out correlation analysis between 10 spamming botnets identified by analyzing 3 weeks of spam emails in our previous work and malware infected hosts that observed at our darknets and honey pots. By comparing members (i.e., bots) of 10 spamming botnets with source hosts of dark net and honey pot traffic, we found that 7.2% ~ 37.5% of spamming botnets has been infected by four different malwares at least.

Application of string kernel based support vector machine for malware packer identification

Packing is among the most popular obfuscation techniques to impede anti-virus scanners from successfully detecting malware. In this paper we propose a string-kernel-based support vector machine classifier to identify the packer that is used to create a given malware program. Our approach is featured by the following characteristics. First, the adoption of a string-kernel-based method bridges the gap between signature-based and machine-learning-base approaches. Second, the kernel function derived from the Levenshtein distance integrates important domain knowledge in the learning process. Then, application of support vector machine, a state-of-the-art classifier, enables an automated packer identification scheme with high generalization ability and time efficiency. Finally, selection of the code segment with the most essential packer relevant information further boosts the classification performance. Experiments on a dataset of 3228 binary programs composed of packed files created by 25 packers show that the proposed approach outperforms PEiD and previous machine-learning-based approaches in prediction accuracy with a large margin. This method can help to improve the scanning efficiency of anti-virus products and promote efficient back-end malware research.

A study on association rule mining of darknet big data

Global darknet monitoring provides an effective way to observe cyber-attacks that are significantly threatening network security and management. In this paper, we present a study on characterization of cyberattacks in the big stream data collected in a large scale distributed darknet using association rule learning. The experiment shows that association rule learning in the darknet stream data can support strategic cyberattack countermeasure in the following ways. First, statistics computed from malware-specific rules can lead to better understanding of the global trend of cyberattacks in the Internet. Second, strong association rules can lead to further insights into the nature of the attacking tools and hence expedite the diagnosis. Then, the discovery of emerging new attacks may lead to early detection and prompt prevention of pandemic incidents, preventing damage to the IT infrastructure and extensive financial loss. Finally, exploring the knowledge in the frequent attacking patterns can enable accurate prediction of future attacks from analyzed hosts, which could improve the performance of honeypot systems to collect more pertinent malware information using limited system and network resources.

DAEDALUS: Novel Application of Large-Scale Darknet Monitoring for Practical Protection of Live Networks

Large-scale darknet monitoring is an effective approach to grasp a global trend of malicious activities on the Internet, such as the world-wide spread of malwares. There, however, have been a gap between the darknet monitoring and actual security operations on live networks, namely the global trend has less direct contribution to protect the live networks. Therefore, we propose a novel application of large-scale darknet monitoring that significantly contributes to the security of live networks. In contrast to the conventional method, wherein the packets received from the outside are observed, we employ a large-scale distributed darknet that consists of several organizations that mutually observe the malicious packets transmitted from the inside of the organizations. Based on this approach, we have developed an alert system called DAEDALUS (direct alert environment for darknet and livenet unified security). We present the primary experimental results obtained from the actual deployment of DAEDALUS.