Kimmo Hätönen

Knowledge discovery from telecommunication network alarm databases

A telecommunication network produces daily large amounts of alarm data. The data contains hidden valuable knowledge about the behavior of the network. This knowledge can be used in filtering redundant alarms, locating problems in, the network, and possibly in predicting severe faults. We describe the TASA (Telecommunication Network Alarm Sequence Analyzer) system for discovering and browsing knowledge from large alarm databases. The system is built on the basis of viewing knowledge discovery as an interactive and iterative process, containing data collection, pattern discovery, rule postprocessing, etc. The system uses a novel framework for locating frequently occurring episodes from sequential data. The TASA system offers a variety of selection and ordering criteria for episodes, and supports iterative retrieval from the discovered knowledge. This means that a large part of the iterative nature of the KDD process can be replaced by iteration in the rule postprocessing stage. The user interface is based on dynamically generated HTML. The system is in experimental use, and the results are encouraging: some of the discovered knowledge is being integrated into the alarm handling software of telecommunication operators.

A computer host-based user anomaly detection system using the self-organizing map

Computer systems are vulnerable to abuse by insiders and to penetration by outsiders. The amount of monitoring data generated in computer networks is enormous. Tools are needed to ease the work of system operators. Anomaly detection attempts to recognize abnormal behavior to detect intrusions. A prototype UNIX anomaly detection system has been constructed. The system is host-based and monitors computer network host users. The system contains an automatic anomaly detection component. This component uses a test based on the self-organizing map to test if user behavior is anomalous. Both the test and the application are presented.

Advanced analysis methods for 3G cellular networks

The operation and maintenance of the third generation (3G) mobile networks will be challenging. These networks will be strongly service driven, and this approach differs significantly from the traditional speech dominated in the second generation (2G) approach. Compared to 2G, in 3G, the mobile cells interact and interfere with each other more, they have hundreds of adjustable parameters, and they monitor and record data related to several hundreds of different variables in each cell. This paper shows that a neural network algorithm called the self-organizing map, together with a conventional clustering method like the k-means, can effectively be used to simplify and focus network analysis. It is shown that these algorithms help in visualizing and grouping similarly behaving cells. Thus, it is easier for a human expert to discern different states of the network. This makes it possible to perform faster and more efficient troubleshooting and optimization of the parameters of the cells. The presented methods are applicable for different radio access network technologies.

TASA: Telecommunication Alarm Sequence Analyzer or how to enjoy faults in your network

Todays large and complex telecommunication networks produce large amounts of alarms daily. The sequence of alarms contains valuable knowledge about the behavior of the network, but much of the knowledge is fragmented and hidden in the vast amount of data. Regularities in the alarms can be used in fault management applications, e.g., for filtering redundant alarms, locating problems in the network, and possibly in predicting severe faults. In this paper we describe TASA (Telecommunication Alarm Sequence Analyzer), a novel system for discovering interesting regularities in the alarms. In the core of the system are algorithms for locating frequent alarm episodes from the alarm stream and presenting them as rules. Discovered rules can then be explored with flexible information retrieval tools that support iteration. The user interface is hypertext, based on HTML, and can be used with a standard WWW browser. TASA is in experimental use and has already discovered rules that have been integrated into the alarm handling software of an operator.

What makes expert systems survive over 10 years—empirical evaluation of several engineering applications

Abstract This case study analyzes eight expert system applications that have successfully been in industrial use for a long time. We have personally been involved in the development of these applications and are thus in a good position to analyze what is important for a successful application and what kind of mistakes can be made. Since the development of the applications started in 1986–1990 and some of them are still in use we are able to observe what has happened to those applications during their lifetime. Our key observations are related to the scope of the applications, to the trade-off between usability and automation, to the role of human experts in the use and development of expert systems, on the technical solutions used, on aspects of the operation of the expert system and on the similarities between expert systems and information systems. The key findings are expressed as 20 hypotheses for successful expert systems. The support of each application to the hypotheses is discussed.

Local anomaly detection for mobile network monitoring

Huge amounts of operation data are constantly collected from various parts of communication networks. These data include measurements from the radio connections and system logs from servers. System operators and developers need robust, easy to use decision support tools based on these data. One of their key applications is to detect anomalous phenomena of the network. In this paper we present an anomaly detection method that describes the normal states of the system with a self-organizing map (SOM) identified from the data. Large deviation in the data samples from the SOM nodes is detected as anomalous behavior. Large deviation has traditionally been detected using global thresholds. If variation of the data occurs in separate parts of the data space, the global thresholds either fail to reveal anomalies or reveal false anomalies. Instead of one global threshold, we can use local thresholds, which depend on the local variation of the data. We also present a method to find an adaptive threshold using the distribution of the deviations. Our anomaly detection method can be used both in exploration of history data or comparison of unforeseen data against a data model derived from history data. It is applicable to wide range of processes that produce multivariate data. In this paper we present examples of this method applied to server log data and radio interface data from mobile networks.

Comprehensive Log Compression with Frequent Patterns

In this paper we present a comprehensive log compression (CLC) method that uses frequent patterns and their condensed representations to identify repetitive information from large log files generated by communications networks. We also show how the identified information can be used to separate and filter out frequently occurring events that hide other, unique or only a few times occurring events. The identification can be done without any prior knowledge about the domain or the events. For example, no pre-defined patterns or value combinations are needed. This separation makes it easier for a human observer to perceive and analyse large amounts of log data. The applicability of the CLC method is demonstrated with real-world examples from data communication networks.

Towards an abstraction layer for security assurance measurements: (invited paper)

Measurement of any complex, operational system is challenging due to the continuous independent evolution of the components. Security risks introduce another dimension of dynamicity, reflected to risk management and security assurance activities. The availability of different measurements and their properties will vary during the overall system lifecycle. To be useful, a measurement framework in this context needs to be able to adapt to both the changes in the target of measurement and in the available measurement infrastructure. In this study, we introduce a taxonomy-based approach for relating the available and attainable measurements to the measurement requirements of security assurance plans by providing an Abstraction Layer that makes it easier to manage these dynamic features. The introduced approach is investigated in terms of a security assurance case example of firewall functionality in a Push E-mail service system.

Architecture for High Confidence Cloud Security Monitoring

Operational security assurance of a networked system requires providing constant and up-to-date evidence of its operational state. In a cloud-based environment we deploy our services as virtual guests running on external hosts. As this environment is not under our full control, we have to find ways to provide assurance that the security information provided from this environment is accurate, and our software is running in the expected environment. In this paper, we present an architecture for providing increased confidence in measurements of such cloud-based deployments. The architecture is based on a set of deployed measurement probes and trusted platform modules (TPM) across both the host infrastructure and guest virtual machines. The TPM are used to verify the integrity of the probes and measurements they provide. This allows us to ensure that the system is running in the expected environment, the monitoring probes have not been tampered with, and the integrity of measurement data provided is maintained. Overall this gives us a basis for increased confidence in the security of running parts of our system in an external cloud-based environment.

Towards Trusted Environment in Cloud Monitoring

This paper investigates the problem of providing trusted monitoring information on a cloud environment to the cloud customers. The general trust between customer and provider is taken as a starting point. The paper discusses possible methods to strengthen this trust. It focuses on establishing a chain of trust inside the provider infrastructure to supply monitoring data for the customer. The goal is to enable delivery of state and event information to parties outside the cloud infrastructure. The current technologies and research are reviewed for the solution and the usage scenario is presented. Based on such technology, higher assurance of the cloud can be presented to the customer. This allows customers with high security requirements and responsibilities to have more confidence in accepting the cloud as their platform of choice.