Mario Couture

A host-based anomaly detection approach by representing system calls as states of kernel modules

Despite over two decades of research, high false alarm rates, large trace sizes and high processing times remain among the key issues in host-based anomaly intrusion detection systems. In an attempt to reduce the false alarm rate and processing time while increasing the detection rate, this paper presents a novel anomaly detection technique based on semantic interactions of system calls. The key concept is to represent system calls as states of kernel modules, analyze the state interactions, and identify anomalies by comparing the probabilities of occurrences of states in normal and anomalous traces. In addition, the proposed technique allows a visual understanding of system behaviour, and hence a more informed decision making. We evaluated this technique on Linux based programs of UNM datasets and a new modern Firefox dataset. We created the Firefox dataset on Linux using contemporary test suites and hacking techniques. The results show that our technique yields fewer false alarms and can handle large traces with smaller (or comparable) processing times compared against the existing techniques for the host based anomaly intrusion detection systems.

An improved Hidden Markov Model for anomaly detection using frequent common patterns

Host-based intrusion detection techniques are needed to ensure the safety and security of software systems, especially, if these systems handle sensitive data. Most host-based intrusion detection systems involve building some sort of reference models offline, usually from execution traces (in the absence of the source code), to characterize the system healthy behavior. The models can later be used as a baseline for online detection of abnormal behavior. Perhaps the most popular techniques are the ones based on the use of Hidden Markov Models (HMM). These techniques, however, require long training time of the models, which makes them computationally infeasible, the main reason being the large size of typical traces. In this paper, we propose an improved HMM using the concept of frequent common patterns. In other words, we build models based on extracting the largest n-grams (patterns) in the traces instead of taking each trace event on its own. We show through a case study that our approach can reduce the training time by 31.96%-48.44% compared to the original HMM algorithms while keeping almost the same accuracy rate.

Total ADS: Automated Software Anomaly Detection System

When a software system starts behaving abnormally during normal operations, system administrators resort to the use of logs, execution traces, and system scanners (e.g., anti-malwares, intrusion detectors, etc.) to diagnose the cause of the anomaly. However, the unpredictable context in which the system runs and daily emergence of new software threats makes it extremely challenging to diagnose anomalies using current tools. Host-based anomaly detection techniques can facilitate the diagnosis of unknown anomalies but there is no common platform with the implementation of such techniques. In this paper, we propose an automated anomaly detection framework (Total ADS) that automatically trains different anomaly detection techniques on a normal trace stream from a software system, raise anomalous alarms on suspicious behaviour in streams of trace data, and uses visualization to facilitate the analysis of the cause of the anomalies. Total ADS is an extensible Eclipse-based open source framework that employs a common trace format to use different types of traces, a common interface to adapt to a variety of anomaly detection techniques (e.g., HMM, sequence matching, etc.). Our case study on a modern Linux server shows that Total ADS automatically detects attacks on the server, shows anomalous paths in traces, and provides forensic insights.

On-device anomaly detection for resource-limited systems

As small-scale embedded systems such as Smartphones rapidly evolve, mobile malwares grow increasingly more sophisticated and dangerous. An important attack vector targeting Android Smartphone is repackaging legitimate applications to inject malicious activities, where such repackaging can be performed before or after the installation of applications on the Smartphone. To detect the behaviour deviation of applications caused by the injected malicious activities, complex anomaly detection algorithms are usually applied, however they require a system resources budget that is beyond the capacities of these small-scale devices. This paper focuses on the usability of on-device anomaly detection algorithms and proposes a detection framework for Android-based devices. The proposed solution allows using a remote server without relying entirely on it. The experimental results allow building resources consumption profiles of the studied anomaly detections algorithms and thus, provide reliable measurements that help define trade-offs between detection accuracy and resource consumption.

Software behaviour correlation in a redundant and diverse environment using the concept of trace abstraction

Redundancy and diversity has been shown to be an effective approach for ensuring service continuity (an important requirement for autonomic systems) despite the presence of anomalies due to attacks or faults. In this paper, we focus on operating system (OS) diversity, which is useful in helping a system survive kernel-level anomalies. We propose an approach for detecting anomalies in the presence of OS diversity. We achieve this by comparing kernel-level traces generated from instances of the same application deployed on different OS. Our trace correlation process relies on the concept of trace abstraction, in which low-level system events are transformed into higher-level concepts, freeing the trace from OS-related events. We show the effectiveness of our approach through a case study, in which we selected Linux and FreeBSD as target OS. We also report on lessons learned, setting the ground for future research.

On the Comparison of User Space and Kernel Space Traces in Identification of Software Anomalies

Corrective software maintenance consumes 30-60% time of software maintenance activities. Automated failure reporting has been introduced to facilitate developers in debugging failures during corrective maintenance. However, reports of software with large user bases overwhelm developers in identification of the origins of faults, and in many cases it is not known whether reports of failures contain information about faults. Prior techniques employ different classification or anomaly detection algorithms on user space traces (e.g., function calls) or kernel space traces (e.g., system calls) to detect anomalies in software behaviour. Each algorithm and type of tracing (user space or kernel space) has its advantages and disadvantages. For example, user space tracing is useful in detailed analysis of anomalous (faulty) behaviour of a program whereas kernel space tracing is useful in identifying system intrusions, program intrusions, or malicious programs even if source program code is different. If one type of tracing or algorithm is infeasible to implement then it is important to know whether we can substitute another type of tracing and algorithm. In this paper, we compare user space and kernel space tracing by employing different types of classification algorithms on the traces of various programs. Our results show that kernel space tracing can be used to identify software anomalies with better accuracy than user space tracing. In fact, the majority of software anomalies (approximately 90%) in a software application can be best identified by using a classification algorithm on kernel space traces.

Monitoring system calls for anomaly detection in modern operating systems

Host-based intrusion detection systems monitor systems in operation for significant deviations from normal (and healthy) behaviour. Many approaches have been proposed in the literature. Most of them, however, do not consider even the basic attack prevention mechanisms that are activated by default on todays many operating systems. Examples of such mechanisms include Address Space Layout Randomization and Data Execution Prevention. With such security methods in place, attackers are forced to perform additional actions to circumvent them. In this research, we conjecture that some of these actions may require the use of additional system calls. If so, one can trace such attacks to discover attack patterns that can later be used to enhance the detection power of anomaly detection systems. The purpose of this short paper is to motivate the need to investigate the impact of attack on system calls while trying to overcome these prevention mechanisms.

Towards a formal framework for evaluating the effectiveness of system diversity when applied to security

N-version programming has been shown to be an effective way to increase the reliability of systems. In this study, we examine the possibility of extending this approach to address security, rather than reliability concerns. We focus specifically on how to evaluate the efficiency of the use of diversity for security. We show that while several key elements must be taken into account when N-version programming is used for security rather than reliability, it is nonetheless possible to devise a reasoning framework to evaluate the efficiency of this development paradigm in a security context. This framework allows us to reason about the most effective way to use diversity for security.