Markus Wurzenberger

Incremental Clustering for Semi-Supervised Anomaly Detection applied on Log Data

Anomaly detection based on white-listing and self-learning has proven to be a promising approach to detect customized and advanced cyber attacks. Anomaly detection aims at detecting significant deviations from normal system and network behavior. A well-known method to classify anomalous and normal system behavior is clustering of log lines. However, this approach has been applied for forensic purposes only, where log data dumps are investigated retrospectively. In order to make this concept applicable for on-line anomaly detection, i.e., at the time the log lines are produced, some major extensions to existing approaches are required. Especially distance based clustering approaches usually fail building the required large distance matrices and rely on time-consuming recalculations of the cluster-map on every arriving log line. An incremental clustering approach seems suitable to solve this issues. Thus, we introduce a semi-supervised concept for incremental clustering of log data that builds the basis for a novel on-line anomaly detection solution based on log data streams. Its operation is independent from the syntax and semantics of the processed log lines, which makes it generally applicable. We demonstrate that that the introduced anomaly detection approach allows to achieve both a high recall and a high precision while maintaining linear complexity.

Discovering Insider Threats from Log Data with High-Performance Bioinformatics Tools

Since the number of cyber attacks by insider threats and the damage caused by them has been increasing over the last years, organizations are in need for specific security solutions to counter these threats. To limit the damage caused by insider threats, the timely detection of erratic system behavior and malicious activities is of primary importance. We observed a major paradigm shift towards anomaly-focused detection mechanisms, which try to establish a baseline of system behavior -- based on system logging data -- and report any deviations from this baseline. While these approaches are promising, they usually have to cope with scalability issues. As the amount of log data generated during IT operations is exponentially growing, high-performance security solutions are required that can handle this huge amount of data in real time. In this paper, we demonstrate how high-performance bioinformatics tools can be leveraged to tackle this issue, and we demonstrate their application to log data for outlier detection, to timely detect anomalous system behavior that points to insider attacks.

Establishing national cyber situational awareness through incident information clustering

The number and type of threats to modern information and communication networks has increased massively in the recent years. Furthermore, the system complexity and interconnectedness has reached a level which makes it impossible to adequately protect networked systems with standard security solutions. There are simply too many unknown vulnerabilities, potential configuration mistakes and therefore enlarged attack surfaces and channels. A promising approach to better secure todays networked systems is information sharing about threats, vulnerabilities and indicators of compromise across organizations; and, in case something went wrong, to report incidents to national cyber security centers. These measures enable early warning systems, support risk management processes, and increase the overall situational awareness of organizations. Several cyber security directives around the world, such as the EU Network and Information Security Directive and the equivalent NIST Framework, demand specifically national cyber security centers and policies for organizations to report on incidents. However, effective tools to support the operation of such centers are rare. Typically, existing tools have been developed with the single organization as customer in mind. These tools are often not appropriate either for the large amounts of data or for the application use case at all. In this paper, we therefore introduce a novel incident clustering model and a system architecture along with a prototype implementation to establish situational awareness about the security of participating organizations. This is a vital prerequisite to plan further actions towards securing national infrastructure assets.

AECID: A Self-learning Anomaly Detection Approach based on Light-weight Log Parser Models.

In recent years, new forms of cyber attacks with an unprecedented sophistication level have emerged. Additionally, systems have grown to a size and complexity so that their mode of operation is barely understandable any more, especially for chronically understaffed security teams. The combination of ever increasing exploitation of zero day vulnerabilities, malware auto-generated from tool kits with varying signatures, and the still problematic lack of user awareness is alarming. As a consequence signature-based intrusion detection systems, which look for signatures of known malware or malicious behavior studied in labs, do not seem fit for future challenges. New, flexibly adaptable forms of intrusion detection systems (IDS), which require just minimal maintenance and human intervention, and rather learn themselves what is considered normal in an infrastructure, are a promising means to tackle today’s serious security situation. This paper introduces ÆCID, a new anomaly-based IDS approach, that incorporates many features motivated by recent research results, including the automatic classification of events in a network, their correlation, evaluation, and interpretation up to a dynamically-configurable alerting system. Eventually, we foresee ÆCID to be a smart sensor for established SIEM solutions. Parts of ÆCID are open source and already included in Debian Linux and Ubuntu. This paper provides vital information on its basic design, deployment scenarios and application cases to support the research community as well as early adopters of the software package.

Time Series Analysis: Unsupervised Anomaly Detection Beyond Outlier Detection

Anomaly detection on log data is an important security mechanism that allows the detection of unknown attacks. Self-learning algorithms capture the behavior of a system over time and are able to identify deviations from the learned normal behavior online. The introduction of clustering techniques enabled outlier detection on log lines independent from their syntax, thereby removing the need for parsers. However, clustering methods only produce static collections of clusters. Therefore, such approaches frequently require a reformation of the clusters in dynamic environments due to changes in technical infrastructure. Moreover, clustering alone is not able to detect anomalies that do not manifest themselves as outliers but rather as log lines with spurious frequencies or incorrect periodicity. In order to overcome these deficiencies, in this paper we introduce a dynamic anomaly detection approach that generates multiple consecutive cluster maps and connects them by deploying cluster evolution techniques. For this, we design a novel clustering model that allows tracking clusters and determining their transitions. We detect anomalous system behavior by applying time-series analysis to relevant metrics computed from the evolving clusters. Finally, we evaluate our solution on an illustrative scenario and validate the achieved quality of the retrieved anomalies with respect to the runtime.

Applying High-Performance Bioinformatics Tools for Outlier Detection in Log Data

Most of todays security solutions, such as security information and event management (SIEM) and signature based IDS, require the operator to evaluate potential attack vectors and update detection signatures and rules in a timely manner. However, todays sophisticated and tailored advanced persistent threats (APT), malware, ransomware and rootkits, can be so complex and diverse, and often use zero day exploits, that a pure signature-based blacklisting approach would not be sufficient to detect them. Therefore, we could observe a major paradigm shift towards anomaly-based detection mechanisms, which try to establish a system behavior baseline - either based on netflow data or system logging data - and report any deviations from this baseline. While these approaches look promising, they usually suffer from scalability issues. As the amount of log data generated during IT operations is exponentially growing, high-performance analysis methods are required that can handle this huge amount of data in real-time. In this paper, we demonstrate how high-performance bioinformatics tools can be applied to tackle this issue. We investigate their application to log data for outlier detection to timely reveal anomalous system behavior that points to cyber attacks. Finally, we assess the detection capability and run-time performance of the proposed approach.

Correlating cyber incident information to establish situational awareness in Critical Infrastructures

Protecting Critical Infrastructures (CIs) against contemporary cyber attacks has become a crucial as well as complex task. Modern attack campaigns, such as Advanced Persistent Threats (APTs), leverage weaknesses in the organizations business processes and exploit vulnerabilities of several systems to hit their target. Although their life-cycle can last for months, these campaigns typically go undetected until they achieve their goal. They usually aim at performing data exfiltration, cause service disruptions and can also undermine the safety of humans. Novel detection techniques and incident handling approaches are therefore required, to effectively protect CIs networks and timely react to this type of threats. Correlating large amounts of data, collected from a multitude of relevant sources, is necessary and sometimes required by national authorities to establish cyber situational awareness, and allow to promptly adopt suitable countermeasures in case of an attack. In this paper we propose three novel methods for security information correlation designed to discover relevant insights and support the establishment of cyber situational awareness.

Complex log file synthesis for rapid sandbox-benchmarking of security- and computer network analysis tools

Today Information and Communications Technology (ICT) networks are a dominating component of our daily life. Centralized logging allows keeping track of events occurring in ICT networks. Therefore a central log store is essential for timely detection of problems such as service quality degradations, performance issues or especially security-relevant cyber attacks. There exist various software tools such as security information and event management (SIEM) systems, log analysis tools and anomaly detection systems, which exploit log data to achieve this. While there are many products on the market, based on different approaches, the identification of the most efficient solution for a specific infrastructure, and the optimal configuration is still an unsolved problem. Todays general test environments do not sufficiently account for the specific properties of individual infrastructure setups. Thus, tests in these environments are usually not representative. However, testing on the real running productive systems exposes the network infrastructure to dangerous or unstable situations. The solution to this dilemma is the design and implementation of a highly realistic test environment, i.e. sandbox solution, that follows a different - novel - approach. The idea is to generate realistic network event sequence (NES) data that reflects the actual system behavior and which is then used to challenge network analysis software tools with varying configurations safely and realistically offline. In this paper we define a model, based on log line clustering and Markov chain simulation to create this synthetic log data. The presented model requires only a small set of real network data as an input to understand the complex real system behavior. Based on the inputs characteristics highly realistic customer specified NES data is generated. To prove the applicability of the concept developed in this work, we conclude the paper with an illustrative example of evaluation and test of an existing anomaly detection system by using generated NES data. HighlightsGenerating log data that reflects realistic network behavior.Log data modeling, based on log line clustering and Markov chain simulation.Rate, analyze and improve software tools, which exploit log data.Detailed evaluation of the model and presentation of an illustrative application.Cornerstones to improve the selection, deployment and operation of IDSs.

Countering targeted cyber-physical attacks using anomaly detection in self-adaptive Industry 4.0 Systems

This paper presents a novel approach to flexibly control the depth of monitoring applied to CPS-enabled safety-critical infrastructures, to timely detect deviations from the desired operational status, and discusses how the application of anomaly detection (AD) techniques can be further leveraged to automatically adapt the security controls of the infrastructure itself.ZusammenfassungDieser Beitrag stellt einen neuartigen Ansatz zur flexiblen Steuerung des Grades der Überwachung in CPS-fähigen sicherheitskritischen Infrastrukturen vor, um Abweichungen vom gewünschten Betriebszustand rechtzeitig zu erkennen, und diskutiert, wie die Anwendung von Anomalie-Erkennungstechniken genutzt werden kann, um die Sicherheitskontrollen der Infrastruktur automatisch anzupassen.

Dynamic log file analysis: An unsupervised cluster evolution approach for anomaly detection

Abstract Technological advances and increased interconnectivity have led to a higher risk of previously unknown threats. Cyber Security therefore employs Intrusion Detection Systems that continuously monitor log lines in order to protect systems from such attacks. Existing approaches use string metrics to group similar lines into clusters and detect dissimilar lines as outliers. However, such methods only produce static views on the data and do not sufficiently incorporate the dynamic nature of logs. Changes of the technological infrastructure therefore frequently require cluster reformations. Moreover, such approaches are not suited for detecting anomalies related to frequencies, periodic alterations and interdependencies of log lines. We therefore propose a dynamic log file anomaly detection methodology that incrementally groups log lines within time windows. Thereby, a novel clustering mechanism establishes links between otherwise isolated collections of clusters. Cluster evolution techniques analyze clusters from neighboring time windows and determine transitions such as splits or merges. A self-learning algorithm then detects anomalies in the temporal behavior of these evolving clusters by analyzing metrics derived from their developments. We apply a prototype in an illustrative scenario consisting of a log file containing known anomalies. We thereby investigate the influences of certain parameters on the detection ability and the runtime. The evaluation of this scenario shows that 61.8% of the dynamic changes of log line clusters are correctly identified, while the false alarm rate is only 0.7%. The ability of efficiently detecting these anomalies while self-adjusting to changes of the system environment suggests the applicability of the introduced approach.