Chuanyi Ji

Anomaly detection in IP networks

Network anomaly detection is a vibrant research area. Researchers have approached this problem using various techniques such as artificial intelligence, machine learning, and state machine modeling. In this paper, we first review these anomaly detection methods and then describe in detail a statistical signal processing technique based on abrupt change detection. We show that this signal processing technique is effective at detecting several network anomalies. Case studies from real network data that demonstrate the power of the signal processing approach to network anomaly detection are presented. The application of signal processing techniques to this area is still in its infancy, and we believe that it has great potential to enhance the field, and thereby improve the reliability of IP networks.

Proactive network fault detection

The increasing role of communication networks in todays society results in a demand for higher levels of network availability and reliability. At the same time, fault management is becoming more difficult due to the dynamic nature and heterogeneity of networks. We propose an intelligent monitoring system using adaptive statistical techniques. The system continually learns the normal behavior of the network and detects deviations from the norm. Within the monitoring system, the measurements are segmented, and features extracted from the segments are used to describe the normal behavior of the measurement variables. This information is combined in the structure of a Bayesian network. The proposed system is thereby able to detect unknown or unseen faults. Experimental results on real network data demonstrate that the proposed system can detect abnormal behavior before a fault actually occurs.

Combinations of weak classifiers

To obtain classification systems with both good generalization performance and efficiency in space and time, we propose a learning method based on combinations of weak classifiers, where weak classifiers are linear classifiers (perceptrons) which can do a little better than making random guesses. A randomized algorithm is proposed to find the weak classifiers. They are then combined through a majority vote. As demonstrated through systematic experiments, the method developed is able to obtain combinations of weak classifiers with good generalization performance and a fast training time on a variety of test problems and real applications. Theoretical analysis on one of the test problems investigated in our experiments provides insights on when and why the proposed method works. In particular, when the strength of weak classifiers is properly chosen, combinations of weak classifiers can achieve a good generalization performance with polynomial space- and time-complexity.

Spatial-temporal modeling of malware propagation in networks

Network security is an important task of network management. One threat to network security is malware (malicious software) propagation. One type of malware is called topological scanning that spreads based on topology information. The focus of this work is on modeling the spread of topological malwares, which is important for understanding their potential damages, and for developing countermeasures to protect the network infrastructure. Our model is motivated by probabilistic graphs, which have been widely investigated in machine learning. We first use a graphical representation to abstract the propagation of malwares that employ different scanning methods. We then use a spatial-temporal random process to describe the statistical dependence of malware propagation in arbitrary topologies. As the spatial dependence is particularly difficult to characterize, the problem becomes how to use simple (i.e., biased) models to approximate the spatially dependent process. In particular, we propose the independent model and the Markov model as simple approximations. We conduct both theoretical analysis and extensive simulations on large networks using both real measurements and synthesized topologies to test the performance of the proposed models. Our results show that the independent model can capture temporal dependence and detailed topology information and, thus, outperforms the previous models, whereas the Markov model incorporates a certain spatial dependence and, thus, achieves a greater accuracy in characterizing both transient and equilibrium behaviors of malware propagation.

Adaptive thresholding for proactive network problem detection

The detection of network fault scenarios has been achieved using the statistical information contained in the Management Information Base (MIB) variables. An appropriate subset of MIB variables was chosen in order to adequately describe the function of the node. The time series data obtained from these variables was analyzed using a sequential generalized likelihood ratio (GLR) test. The GLR test was used to detect the change points in the behavior of the variables. Using a binary hypothesis test, variable level alarms were generated based on the magnitude of the detected changes as compared to the normal situation. These alarms were combined using a duration filter resulting in a set of node level alarms, which correlated with the experimentally observed network faults and performance problems. The algorithm has been tested on real network data. The applicability of our algorithm to a heterogeneous node was confirmed by using the MIB data from a second node. Interestingly, for most of the faults studied, detection occurs in advance of the fault (at least 5 min) and the algorithm is simple enough for potential online implementation: thus allowing the possibility of prediction and recovery in the future.

Spatial-Temporal Characteristics of Internet Malicious Sources

This paper presents a large scale longitudinal study of the spatial and temporal features of malicious source addresses. The basis of our study is a 402-day trace of over 7 billion Internet intrusion attempts provided by DShield.org, which includes 160 million unique source addresses. Specifically, we focus on spatial distributions and temporal characteristics of malicious sources. First, we find that one out of 27 hosts is potentially a scanning source among 232 IPv4 addresses. We then show that malicious sources have a persistent, non-uniform spatial distribution. That is, more than 80% of the sources send packets from the same 20% of the IPv4 address space over time. We also find that 7.3% of malicious source addresses are unroutable, and that some source addresses are correlated. Next, we show that most sources have a short lifetime. 57.9 % of the source addresses appear only once in the trace, and 90% of source addresses appear less than 5 times. These results have implications for both attacks and defenses.

Intelligent agents for proactive fault detection

As the Internet becomes a critical component of our society, a key challenge is to maintain network availability and reliability. Intelligent processing agents that reside at network nodes use an adaptive learning method to detect abnormal network behavior before a fault actually occurs. In a test at Rensselaer Polytechnic Institute, an agent on a router detected a file server failure 12 minutes before it occurred.

Anomaly Detection Approaches for Communication Networks

In recent years, network anomaly detection has become an important area for both commercial interests as well as academic research. Applications of anomaly detection typically stem from the perspectives of network monitoring and network security. In network monitoring, a service provider is often interested in capturing such network characteristics as heavy flows, flow size distributions, and the number of distinct flows. In network security, the interest lies in characterizing known or unknown anomalous patterns of an attack or a virus.

Measuring Network-Aware Worm Spreading Ability

This work investigates three aspects: (a) a network vulnerability as the non-uniform vulnerable-host distribution, (b) threats, i.e., intelligent worms that exploit such a vulnerability, and (c) defense, i.e., challenges for fighting the threats. We first study five data sets and observe consistent clustered vulnerable-host distributions. We then present a new metric, referred to as the non-uniformity factor, which quantifies the unevenness of a vulnerable-host distribution. This metric is essentially the Renyi information entropy and better characterizes the non-uniformity of a distribution than the Shannon entropy. We then analytically and empirically measure the infection rate and the propagation speed of network-aware worms. We show that a representative network-aware worm can increase the spreading speed by exactly or nearly a non-uniformity factor when compared to a random-scanning worm at the early stage of worm propagation. This implies that when a worm exploits an uneven vulnerable-host distribution as a network-wide vulnerability, the Internet can be infected much more rapidly. Furthermore, we analyze the effectiveness of defense strategies on the spread of network-aware worms. Our results demonstrate that counteracting network-aware worms is a significant challenge for the strategies that include host-based defense and IPv6.

Understanding Localized-Scanning Worms

Localized scanning is a simple technique used by attackers to search for vulnerable hosts. Localized scanning trades off between the local and the global search of vulnerable hosts and has been used by Code Red II and Ninida worms. As such a strategy is so simple yet effective in attacking the Internet, it is important that defenders understand the spreading ability and behaviors of localized-scanning worms. In this work, we first characterize the relationships between vulnerable-host distributions and the spread of localized-scanning worms through mathematical modeling and analysis, and compare random scanning with localized scanning. We then design an optimal localized-scanning strategy, which provides an upper bound on the spreading speed of localized-scanning self-propagating codes. Furthermore, we construct three variants of localized scanning. Specifically, the feedback localized scanning and the ping-pong localized scanning adapt the scanning methods based on the feedback from the probed host, and thus spread faster than the original localized scanning and meanwhile have a smaller variance.