Kalle Ring Burbeck

ADWICE – anomaly detection with real-time incremental clustering

Anomaly detection, detection of deviations from what is considered normal, is an important complement to misuse detection based on attack signatures. Anomaly detection in real-time places hard requirements on the algorithms used, making many proposed data mining techniques less suitable. ADWICE (Anomaly Detection With fast Incremental Clustering) uses the first phase of the existing BIRCH clustering framework to implement fast, scalable and adaptive anomaly detection. We extend the original clustering algorithm and apply the resulting detection mechanism for analysis of data from IP networks. The performance is demonstrated on the KDD data set as well as on data from a test network at a telecom company. Our experiments show a good detection quality (95 %) and acceptable false positives rate (2.8 %) considering the online, real-time characteristics of the algorithm. The number of alarms is then further reduced by application of the aggregation techniques implemented in the Safeguard architecture.

Scale-up and performance studies of three agent platforms

With maturing technology agents are now a viable choice for distributed computing, also for systems with requirements on dependability and scalability. Agent platforms provide common services to applications developed as agents. Given the abundance of available platforms it is not easy to select an agent platform given a set of applications requirements. Evaluations of relevant properties of agent platforms are therefore needed, but unfortunately few up-to-date evaluations exist. In this paper we introduce and evaluate the three recent agent platforms JADE, Tryllian and SAP. Focus of the evaluation is the important properties of performance, security and scalability. We conclude that all platforms perform very well, but that platform architecture heavily influences the performance.

Adaptive real-time anomaly detection with incremental clustering

Anomaly detection in information (IP) networks, detection of deviations from what is considered normal, is an important complement to misuse detection based on known attack descriptions. Performing anomaly detection in real-time places hard requirements on the algorithms used. First, to deal with the massive data volumes one needs to have efficient data structures and indexing mechanisms. Secondly, the dynamic nature of todays information networks makes the characterisation of normal requests and services difficult. What is considered as normal during some time interval may be classified as abnormal in a new context, and vice versa. These factors make many proposed data mining techniques less suitable for real-time intrusion detection. In this paper we present ADWICE, Anomaly Detection With fast Incremental Clustering, and propose a new grid index that is shown to improve detection performance while preserving efficiency in search. Moreover, we propose two mechanisms for adaptive evolution of the normality model: incremental extension with new elements of normal behaviour, and a new feature that enables forgetting of outdated elements of normal behaviour. These address the needs of a dynamic network environment such as a telecom management network. We evaluate the technique for network-based intrusion detection, using the KDD data set as well as on data from a telecom IP test network. The experiments show good detection quality and act as proof of concept for adaptation of normality.

Alarm Reduction and Correlation in Intrusion Detection Systems

Large Critical Complex Infrastructures are increasingly dependent on IP networks. Reliability by redundancy and tolerance are an imperative for such dependable networks. In order to achieve the desired reliability, the detection of faults, misuse, and attacks is essential. This can be achieved by applying methods of intrusion detection. However, in large systems, these methods produce an uncontrollable vast amount of data which overwhelms human operators. This paper studies the role of alarm reduction and correlation in existing networks for building more intelligent safeguards that support and complement the decisions by the operator. We present an architecture that incorporates Intrusion Detection Systems as sensors, and provides quantitatively and qualitatively improved alarms to the human operator. Alarm reduction via static and adaptive filtering, aggregation, and correlation is demonstrated using realistic data from sensors such as Snort, Samhain, and Syslog.

Alarm reduction and correlation in defence of IP networks

Societys critical infrastructures are increasingly dependent on IP networks. Intrusion detection and tolerance within data networks is therefore imperative for dependability in other domains such as telecommunications and future energy management networks. Todays data networks are protected by human operators who are overwhelmed by the massive information overload through false alarm rates of the protection mechanisms. This paper studies the role of alarm reduction and correlation in supporting the security administrator in an enterprise network. We present an architecture that incorporates intrusion detection systems as sensors, and provides improved alarm data to the human operator or to automated actuators. Alarm reduction and correlation via static and adaptive filtering, normalisation, and aggregation is demonstrated on the output from three sensors (Snort, Samhain and Syslog) used in a telecom test network.

Current research and use of anomaly detection

Anomaly detection in IP networks, detection of deviations from what is considered normal, is an important complement to misuse detection based on known attack descriptions. Anomaly detection is at present time often implemented to some extent in available intrusion detection products. Still much effort is spent on anomaly detection research and many problems remains to be explored. Performing anomaly detection in real-time places hard requirements on the algorithms used. First, to deal with the massive data volumes one needs to have efficient data structures and indexing mechanisms. Secondly, the dynamic nature of todays information networks makes the characterization of normal requests and services difficult. What is considered as normal during some time interval may be classified as abnormal in a new context, and vice versa. These factors make many proposed data mining techniques less suitable for real-time intrusion detection. ADWICE (anomaly detection with fast incremental clustering) uses incremental clustering and an integrated grid-based index to implement fast, scalable and adaptive anomaly detection.

Adaptive real-time anomaly detection with improved index and ability to forget

Anomaly detection in IP networks, detection of deviations from what is considered normal, is an important complement to misuse detection based on known attack descriptions. Performing anomaly detection in real-time places hard requirements on the algorithms used. First, to deal with the massive data volumes one needs to have efficient data structures and indexing mechanisms. Secondly, the dynamic nature of todays information networks makes the characterization of normal requests and services difficult. What is considered as normal during some time interval may be classified as abnormal in a new context, and vice versa. These factors make many proposed data mining techniques less suitable for real-time intrusion detection. In this paper we extend ADW ICE, anomaly detection with fast incremental clustering. Accuracy of ADW ICE classifications is improved by introducing a new grid-based index, and its ability to build models incrementally is extended by introducing forgetting. We evaluate the technique on the KDD data set as well as on data from a real (telecom) IP test network. The experiments show good detection quality and illustrate the usefulness of adapting to normality.