Nong Ye

Multivariate statistical analysis of audit trails for host-based intrusion detection

Intrusion detection complements prevention mechanisms, such as firewalls, cryptography, and authentication, to capture intrusions into an information system while they are acting on the information system. Our study investigates a multivariate quality control technique to detect intrusions by building a long-term profile of normal activities in information systems (norm profile) and using the norm profile to detect anomalies. The multivariate quality control technique is based on Hotellings T/sup 2/ test that detects both counterrelationship anomalies and mean-shift anomalies. The performance of the Hotellings T/sup 2/ test is examined on two sets of computer audit data: a small data set and a large multiday data set. Both data sets contain sessions of normal and intrusive activities. For the small data set, the Hotellings T/sup 2/ test signals all the intrusion sessions and produces no false alarms for the normal sessions. For the large data set, the Hotellings T/sup 2/ test signals 92 percent of the intrusion sessions while producing no false alarms for the normal sessions. The performance of the Hotellings T/sup 2/ test is also compared with the performance of a more scalable multivariate technique-a chi-squared distance test.

Probabilistic techniques for intrusion detection based on computer audit data

This paper presents a series of studies on probabilistic properties of activity data in an information system for detecting intrusions into the information system. Various probabilistic techniques of intrusion detection, including decision tree, Hotellings T/sup 2/ test, chi-square multivariate test, and Markov chain are applied to the same training set and the same testing set of computer audit data for investigating the frequency property and the ordering property of computer audit data. The results of these studies provide answers to several questions concerning which properties are critical to intrusion detection. First, our studies show that the frequency property of multiple audit event types in a sequence of events is necessary for intrusion detection. A single audit event at a given time is not sufficient for intrusion detection. Second, the ordering property of multiple audit events provides additional advantage to the frequency property for intrusion detection. However, unless the scalability problem of complex data models taking into account the ordering property of activity data is solved, intrusion detection techniques based on the frequency property provide a viable solution that produces good intrusion detection performance with low computational overhead.

Robustness of the Markov-chain model for cyber-attack detection

Cyber-attack detection is used to identify cyber-attacks while they are acting on a computer and network system to compromise the security (e.g., availability, integrity, and confidentiality) of the system. This paper presents a cyber-attack detection technique through anomaly-detection, and discusses the robustness of the modeling technique employed. In this technique, a Markov-chain model represents a profile of computer-event transitions in a normal/usual operating condition of a computer and network system (a norm profile). The Markov-chain model of the norm profile is generated from historic data of the systems normal activities. The observed activities of the system are analyzed to infer the probability that the Markov-chain model of the norm profile supports the observed activities. The lower probability the observed activities receive from the Markov-chain model of the norm profile, the more likely the observed activities are anomalies resulting from cyber-attacks, and vice versa. This paper presents the learning and inference algorithms of this anomaly-detection technique based on the Markov-chain model of a norm profile, and examines its performance using the audit data of UNIX-based host machines with the Solaris operating system. The robustness of the Markov-chain model for cyber-attack detection is presented through discussions & applications. To apply the Markov-chain technique and other stochastic process techniques to model the sequential ordering of events, the quality of activity-data plays an important role in the performance of intrusion detection. The Markov-chain technique is not robust to noise in the data (the mixture level of normal activities and intrusive activities). The Markov-chain technique produces desirable performance only at a low noise level. This study also shows that the performance of the Markov-chain techniques is not always robust to the window size: as the window size increases, the amount of noise in the window also generally increases. Overall, this study provides some support for the idea that the Markov-chain technique might not be as robust as the other intrusion-detection methods such as the chi-square distance test technique , although it can produce better performance than the chi-square distance test technique when the noise level of the data is low, such as the Mill & Pascal data in this study.

Computer intrusion detection through EWMA for autocorrelated and uncorrelated data

Reliability and quality of service from information systems has been threatened by cyber intrusions. To protect information systems from intrusions and thus assure reliability and quality of service, it is highly desirable to develop techniques that detect intrusions. Many intrusions manifest in anomalous changes in intensity of events occurring in information systems. In this study, we apply, test, and compare two EWMA techniques to detect anomalous changes in event intensity for intrusion detection: EWMA for autocorrelated data and EWMA for uncorrelated data. Different parameter settings and their effects on performance of these EWMA techniques are also investigated to provide guidelines for practical use of these techniques.

Connectivity distribution and attack tolerance of general networks with both preferential and random attachments

A general class of growing networks is constructed with both preferential and random attachments, which includes random and scale-free networks as limiting cases when a physical parameter is tuned. Formulas are derived characterizing the evolution and distribution of the connectivity, which are verified by numerical computations. Study of the effect of random failures and intentional attacks on the performance of network suggests that general networks which are neither completely random nor scale-free are desirable.

Recent developments in chaotic time series analysis

In this paper, two issues are addressed: (1) the applicability of the delay-coordinate embedding method to transient chaotic time series analysis, and (2) the Hilbert transform methodology for chao...

EWMA forecast of normal system activity for computer intrusion detection

Intrusions into computer systems have caused many quality/reliability problems. Detecting intrusions is an important part of assuring the quality/reliability of computer systems by quickly detecting intrusions and associated quality/reliability problems in order to take corrective actions. In this paper, we present and compare two methods of forecasting normal activities in computer systems for intrusion detection. One forecasting method uses the average of long-term normal activities as the forecast. Another forecasting method uses the EWMA (exponentially weighted moving average) one-step-ahead forecast. We use a Markov chain model to learn and predict normal activities used in the EWMA forecasting method. A forecast of normal activities is used to detect a large deviation of the observed activities from the forecast as a possible intrusion into computer systems. A Chi square distance metric is used to measure the deviation of the observed activities from the forecast of normal activities. The two forecasting methods are tested on computer audit data of normal and intrusive activities for intrusion detection. The results indicate that the Chi square distance measure with the EWMA forecasting provides better performance in intrusion detection than that with the average-based forecasting method.

A supervised clustering and classification algorithm for mining data with mixed variables

This paper presents a data mining algorithm based on supervised clustering to learn data patterns and use these patterns for data classification. This algorithm enables a scalable incremental learning of patterns from data with both numeric and nominal variables. Two different methods of combining numeric and nominal variables in calculating the distance between clusters are investigated. In one method, separate distance measures are calculated for numeric and nominal variables, respectively, and are then combined into an overall distance measure. In another method, nominal variables are converted into numeric variables, and then a distance measure is calculated using all variables. We analyze the computational complexity, and thus, the scalability, of the algorithm, and test its performance on a number of data sets from various application domains. The prediction accuracy and reliability of the algorithm are analyzed, tested, and compared with those of several other data mining algorithms.

Toward Development of Adaptive Service-Based Software Systems

The rapid adoption of service-oriented architecture (SOA) in many large-scale distributed applications requires the development of adaptive service-based software systems (ASBS) with the capability of monitoring the changing system status, analyzing, and controlling tradeoffs among various quality-of-service (QoS) aspects, and adapting service configurations to satisfy multiple QoS requirements simultaneously. In this paper, our results toward the development of adaptive service-based software systems are presented. The formulation of activity-state-QoS (ASQ) models and how to use the data from controlled experiments to establish ASQ models for capturing the cause-effect dynamics among service activities, system resource states, and QoS in service-based systems are presented. Then, QoS monitoring modules based on ASQ models and SOA-compliant simulation models are developed to support the validation of the ASBS design. The main idea for developing QoS adaptation modules based on ASQ models is discussed. An experiment based on a voice communication service is used to illustrate our results.

Job scheduling methods for reducing waiting time variance

Minimizing Waiting Time Variance (WTV) is a job scheduling problem where we schedule a batch of n jobs, for servicing on a single resource, in such a way that the variance of their waiting times is minimized. Minimizing WTV is a well known scheduling problem, important in providing Quality of Service (QoS) in many industries. Minimizing the variance of job waiting times on computer networks can lead to stable and predictable network performance. Since the WTV minimization problem is NP-hard, we develop two heuristic job scheduling methods, called Balanced Spiral and Verified Spiral, which incorporate certain proven properties of optimal job sequences for this problem. We test and compare our methods with four other job scheduling methods on both small and large size problem instances. Performance results show that Verified Spiral gives the best performance for the scheduling methods and problems tested in this study. Balanced Spiral produces comparable results, but at less computational cost. During our investigation we discovered a consistent pattern in the plot of WTV over mean of all possible sequences for a set of jobs, which can be used to evaluate the sacrifice of mean waiting time while pursuing WTV minimization. 2005 Elsevier Ltd. All rights reserved.