Shoichiro Asano

A Study on Detecting Network Anomalies Using Sampled Flow Statistics

We investigate how to detect network anomalies using flow statistics obtained through packet sampling. First, we show that network anomalies generating a huge number of small flows, such as network scans or SYN flooding, become difficult to detect when we execute packet sampling. This is because such flows are more unlikely to be sampled than normal flows. As a solution to this problem, we then show that spatially partitioning the monitored traffic into groups and analyzing the traffic of individual groups can increase the detectability of such anomalies. We also show the effectiveness of the partitioning method using network measurement data

Effect of sampling rate and monitoring granularity on anomaly detectability

In this paper, we quantitatively evaluate how sampling decreases the detectability of anomalous traffic. We build equations to calculate the false positive ratio (FPR) and false negative ratio (FNR) for given values of the sampling rate, statistics of normal traffic, and volume of anomalies to be detected. We show that by changing the measurement granularity, we can detect anomalies even with a low sampling rate and give the equation to derive optimal granularity by using the relationship between the mean and variance of aggregated flows. With those equations, we can answer for the practical questions that arise in actual network operations; what sampling rate to set in order to find the given volume of anomaly, or, if the sampling is too high for actual operation, then what granularity is optimal to find the anomaly for a given lower limit of sampling rate.

Traffic measurement and analysis in an ATM-based internet backbone

This paper reports our measurements and analysis of traffic characteristics in an Internet backbone ATM network. In order to utilize network resource efficiently while satisfying the quality of service requirement, it is important to understand the traffic characteristics. We therefore monitored the traffic from the flow or application level to the cell level on a link between NTTs Open Computer Network (OCN) and the Science Information Network (SINET), which are two of the largest Internet backbone ATM-based networks in Japan. Using the monitored traffic, we also evaluated the performance of the aggregate traffic by real-time simulation. Results show that the performance (cell loss ratio) greatly depended not only on link utilization but also on the number of flows, flow size, and traffic composition in terms of applications. We also found that the degree of self-similarity in the Internet backbone was not large. In addition, we clarified that more statistical multiplexing gain could be obtained in the Internet backbone when more flows were multiplexed onto a link.

Detection accuracy of network anomalies using sampled flow statistics

We investigated the detection accuracy of network anomalies when using flow statistics obtained through packet sampling. Through a case study based on measurement data, we showed that network anomalies generating a large number of small flows, such as network scans or SYN flooding, become difficult to detect during packet sampling. We then developed an analytical model that enables us to quantitatively evaluate the effect of packet sampling and traffic conditions, such as anomalous traffic volume, on detection accuracy. We also investigated how the detection accuracy worsens when the packet sampling rate decreases. In addition, we show that, even with a low sampling rate, spatially partitioning monitored traffic into groups makes it possible to increase detection accuracy. We also developed a method of determining an appropriate number of partitioned groups, and we show its effectiveness. Copyright

Detection of TCP Performance Degradation Using Link Utilization Statistics

In this paper, we propose a method of detecting TCP performance degradation using only bottleneck-link utilization statistics. The variance of link utilization normally increases as the mean link-utilization increases. However, because link-utilization has a maximum of 100%, as the mean approaches 100%, the variance decreases to zero. In this paper, using the M/G/R processor sharing model, we relate this phenomenon to the behavior of flows. We also show that by using this relationship, we can detect TCP performance degradation using the mean and variance of link utilization. Particularly, with this method, a network operator can determine whether or not the degradation originates from the congestion of his/her own network. Because our method requires us to measure link utilization only, the cost of performance management can be greatly decreased compared with the conventional method, which requires dedicated equipment to measure the network performance.

Field Trial of All-Optical 2R Regeneration in 40-Gbit/s WDM Transmission Systems with Optical Add/Drop Multiplexing

Optical signal processing is one of essential technologies for improving the flexibility of all-optical network. Above all, recently there have been a lot of studies regarding all-optical 2R/3R regeneration technology. However, there are few studies about all-optical 2R/3R technologies that are carried out in field environment. In this paper, we report the successful results of field trials of an all-optical 2R regeneration system based on an electro-absorption modulator for 40-Gbit/s WDM transmission systems with optical add/drop multiplexing. It was made sure that by applying the all-optical 2R regeneration system to the optical add/drop multiplexer in the 320-km-long transmission systems the transmission characteristics of the express signal after 320-km transmission and those of the dropped signal at 160-km can be made nearly the same. It is quite important that the transmission characteristics are equal for both the dropped and express channel from a point of view of the system design, and the results in this paper suggests one possible solution for this matter.

On the impact of time scales on tail behavior of long-range dependent Internet traffic

Conventionally, Internet traffic has been modeled using classical Poisson-based models. More recent studies have proposed fractal models such as fractional Brownian motion. However, due to its simplicity, fractional Brownian motion is only efficient for approximating the performance of a class of exactly self-similar traffic, whose correlation property can be described by a single Hurst parameter. In this paper, we examine the tail behavior of long-range dependent Internet traffic, which has a more general correlation property. We propose an analytical method by focusing on the impact of time scales on queueing performance. The properties of traffic data are extracted from traffic traces of real networks, such as a wide area backbone network and a LAN. Results produced by simulation using real traffic data are compared with analytical results obtained by our method.

Detection Accuracy of Network Anomalies Using Sampled Flow Statistics

We investigate the detection accuracy of network anomalies when we use flow statistics obtained through packet sampling. We have already shown, through a case study based on measurement data, that network anomalies generating a huge number of small flows, such as network scans or SYN flooding, become hard to detect when we perform packet sampling. In this paper, we first develop an analytical model that enables us to quantitatively evaluate the effect of packet sampling on the detection accuracy and then investigate why detection accuracy worsens when the packet sampling rate decreases. In addition, we show that, even with a low sampling rate, spatially partitioning the monitored traffic into groups makes it possible to increase the detection accuracy. We also develop a method of determining an appropriate number of partitioned groups and show its effectiveness.

Field trial of automatic chromatic dispersion compensation for 40-Gb/s-based wavelength path protection

We report the results of the field trial of generalized multiprotocol label switching-based 40-Gb/s wavelength path protection by photonic cross-connect. In the trial, an automatic chromatic dispersion (CD) compensator was applied to maintain the quality of a signal after protection operation to compensate for the residual CD along the wavelength path. From the results of the field trial, we have made sure that the automatic CD compensation system is essential for the future all-optical wavelength path network.

Finding Cardinality Heavy-Hitters in Massive Traffic Data and Its Application to Anomaly Detection

We propose an algorithm for finding heavy hitters in terms of cardinality (the number of distinct items in a set) in massive traffic data using a small amount of memory. Examples of such cardinality heavy-hitters are hosts that send large numbers of flows, or hosts that communicate with large numbers of other hosts. Finding these hosts is crucial to the provision of good communication quality because they significantly affect the communications of other hosts via either malicious activities such as worm scans, spam distribution, or botnet control or normal activities such as being a member of a flash crowd or performing peer-to-peer (P2P) communication. To precisely determine the cardinality of a host we need tables of previously seen items for each host (e. g., flow tables for every host) and this may infeasible for a high-speed environment with a massive amount of traffic. In this paper, we use a cardinality estimation algorithm that does not require these tables but needs only a little information called the cardinality summary. This is made possible by relaxing the goal from exact counting to estimation of cardinality. In addition, we propose an algorithm that does not need to maintain the cardinality summary for each host, but only for partitioned addresses of a host. As a result, the required number of tables can be significantly decreased. We evaluated our algorithm using actual backbone traffic data to find the heavy-hitters in the number of flows and estimate the number of these flows. We found that while the accuracy degraded when estimating for hosts with few flows, the algorithm could accurately find the top-100 hosts in terms of the number of flows using a limited-sized memory. In addition, we found that the number of tables required to achieve a pre-defined accuracy increased logarithmically with respect to the total number of hosts, which indicates that our method is applicable for large traffic data for a very large number of hosts. We also introduce an application of our algorithm to anomaly detection. With actual traffic data, our method could successfully detect a sudden network scan.