Jan Stiborek

An empirical comparison of botnet detection methods

The results of botnet detection methods are usually presented without any comparison. Although it is generally accepted that more comparisons with third-party methods may help to improve the area, few papers could do it. Among the factors that prevent a comparison are the difficulties to share a dataset, the lack of a good dataset, the absence of a proper description of the methods and the lack of a comparison methodology. This paper compares the output of three different botnet detection methods by executing them over a new, real, labeled and large botnet dataset. This dataset includes botnet, normal and background traffic. The results of our two methods (BClus and CAMNEP) and BotHunter were compared using a methodology and a novel error metric designed for botnet detections methods. We conclude that comparing methods indeed helps to better estimate how good the methods are, to improve the algorithms, to build better datasets and to build a comparison methodology.

Adaptive Multiagent System for Network Traffic Monitoring

Individual anomaly-detection methods for monitoring computer network traffic have relatively high error rates. An agent-based trust-modeling system fuses anomaly data and progressively improves classification to achieve acceptable error rates.

Runtime Monitoring and Dynamic Reconfiguration for Intrusion Detection Systems

Our work proposes a generic architecture for runtime monitoring and optimization of IDS based on the challenge insertion. The challenges, known instances of malicious or legitimate behavior, are inserted into the network traffic represented by NetFlow records, processed with the current traffic and the systems response to the challenges is used to determine its effectiveness and to fine-tune its parameters. The insertion of challenges is based on the threat models expressed as attack trees with attached risk/loss values. The use of threat model allows the system to measure the expected undetected loss and to improve its performance with respect to the relevant threats, as we have verified in the experiments performed on live network traffic.

Game-theoretic approach to adversarial plan recognition

We argue that the problem of adversarial plan recognition, where the observed agent actively tries to avoid detection, should be modeled in the game theoretic framework. We define the problem as an imperfect-information extensive-form game between the observer and the observed agent. We propose a novel algorithm that approximates the optimal solution in the game using Monte-Carlo sampling. The experimental evaluation is performed on a synthetic domain inspired by a network security problem. The proposed method produces significantly better results than several simple baselines on a practically large domain.

Threat-model-driven runtime adaptation and evaluation of intrusion detection system

We present a mechanism for autonomous self-adaptation of a network-based intrusion detection system (IDS). The system is composed of a set of cooperating agents, each of which is based on an existing network behavior analysis method. The self adaptation mechanism is based on the insertion of a small number of challenges, i.e. known instances of past legitimate or malicious behavior. The response of individual system components to these challenges is used to measure and eventually optimize the system performance in terms of accuracy. In this work we show how to choose the challenges in a way such that the IDS attaches more importance to the detection of attacks that cause much damage.

Game theoretical adaptation model for intrusion detection system

We present a self-adaptation mechanism for Network Intrusion Detection System which uses a game-theoretical mechanism to increase system robustness against targeted attacks on IDS adaptation. We model the adaptation process as a strategy selection in sequence of single stage, two player games. The key innovation of our approach is a secure runtime game definition and numerical solution and real-time use of game solutions for dynamic system reconfiguration. Our approach is suited for realistic environments where we typically lack any ground truth information regarding traffic legitimacy/maliciousness and where the significant portion of system inputs may be shaped by the attacker in order to render the system ineffective. Therefore, we rely on the concept of challenge insertion: we inject a small sample of simulated attacks into the unknown traffic and use the system response to these attacks to define the game structure and utility functions. This approach is also advantageous from the security perspective, as the manipulation of the adaptive process by the attacker is far more difficult. Our experimental results suggest that the use of game-theoretical mechanism comes with little or no penalty when compared to traditional self-adaptation methods.

Using Behavioral Similarity for Botnet Command-and-Control Discovery

Malware authors and operators typically collaborate to achieve the optimal profit. They also frequently change their behavior and resources to avoid detection. The authors propose a social similarity metrics that exploits these relationships to improve the effectiveness and stability of the threat propagation algorithm typically used to discover malicious collaboration. Furthermore, they propose behavioral modeling as a way to group similarly behaving servers, enabling extension of the ground truth thats so expensive to obtain in the field of network security. The authors also show that seeding the threat propagation algorithm from a set of coherently behaving servers (instead of from a single known malicious server identified by threat intelligence) makes the algorithm far more effective and significantly more robust, without compromising the precision of findings.

Multiple instance learning for malware classification

This work addresses classification of unknown binaries executed in sandbox by modeling their interaction with system resources (files, mutexes, registry keys and communication with servers over the network) and error messages provided by the operating system, using vocabulary-based method from the multiple instance learning paradigm. It introduces similarities suitable for individual resource types that combined with an approximative clustering method efficiently group the system resources and define features directly from data. This approach effectively removes randomization often employed by malware authors and projects samples into low-dimensional feature space suitable for common classifiers. An extensive comparison to the state of the art on a large corpus of binaries demonstrates that the proposed solution achieves superior results using only a fraction of training samples. Moreover, it makes use of a source of information different than most of the prior art, which increases the diversity of tools detecting the malware, hence making detection evasion more difficult.

On the Value of Coordination in Distributed Self-Adaptation of Intrusion Detection System

We present an empirical study of distributed adaptation in an Intrusion Detection System. The adaptation model is based on a game-theoretical approach and we use regret minimization techniques to find globally robust behavior. We compare the effectiveness of global optimization, when all system components adopt the globally optimized strategy in a synchronized manner, with a fully distributed approach when two layers in the system adapt their strategies as a result of local adaptation process, with no synchronization or signaling. We show that the use of regret minimization techniques results in stable and long-term optimized behavior in both cases. Our experiments were performed on CAMNEP, an intrusion detection system based on analysis of Net Flow data, and were performed on the university network over one month.

Probabilistic analysis of dynamic malware traces

Abstract We propose a method to automatically group unknown binaries executed in sandbox according to their interaction with system resources (files on the filesystem, mutexes, registry keys, network communication with remote servers and error messages generated by operating system) such that each group corresponds to a malware family. The method utilizes probabilistic generative model (Bernoulli mixture model), which allows human-friendly prioritization of identified clusters and extraction of readable behavioral indicators to maximize interpretability. We compare it to relevant prior art on a large set of malware binaries where a quality of cluster prioritization and automatic extraction of indicators of compromise is demonstrated. The proposed approach therefore implements complete pipeline which has the potential to significantly speed-up analysis of unknown samples.