Ashwin Sridharan

Is sampled data sufficient for anomaly detection

Sampling techniques are widely used for traffic measurements at high link speed to conserve router resources. Traditionally, sampled traffic data is used for network management tasks such as traffic matrix estimations, but recently it has also been used in numerous anomaly detection algorithms, as security analysis becomes increasingly critical for network providers. While the impact of sampling on traffic engineering metrics such as flow size and mean rate is well studied, its impact on anomaly detection remains an open question.This paper presents a comprehensive study on whether existing sampling techniques distort traffic features critical for effective anomaly detection. We sampled packet traces captured from a Tier-1 IP-backbone using four popular methods: random packet sampling, random flow sampling, smart sampling, and sample-and-hold. The sampled data is then used as input to detect two common classes of anomalies: volume anomalies and port scans. Since it is infeasible to enumerate all existing solutions, we study three representative algorithms: a wavelet-based volume anomaly detection and two portscan detection algorithms based on hypotheses testing. Our results show that all the four sampling methods introduce fundamental bias that degrades the performance of the three detection schemes, however the degradation curves are very different. We also identify the traffic features critical for anomaly detection and analyze how they are affected by sampling. Our work demonstrates the need for better measurement techniques, since anomaly detection operates on a drastically different information region, which is often overlooked by existing traffic accounting methods that target heavy-hitters.

Mobile call graphs: beyond power-law and lognormal distributions

We analyze a massive social network, gathered from the records of a large mobile phone operator, with more than a million users and tens of millions of calls. We examine the distributions of the number of phone calls per customer; the total talk minutes per customer; and the distinct number of calling partners per customer. We find that these distributions are skewed, and that they significantly deviate from what would be expected by power-law and lognormal distributions. To analyze our observed distributions (of number of calls, distinct call partners, and total talk time), we propose PowerTrack , a method which fits a lesser known but more suitable distribution, namely the Double Pareto LogNormal (DPLN) distribution, to our data and track its parameters over time. Using PowerTrack , we find that our graph changes over time in a way consistent with a generative process that naturally results in the DPLN distributions we observe. Furthermore, we show that this generative process lends itself to a natural and appealing social wealth interpretation in the context of social networks such as ours. We discuss the application of those results to our model and to forecasting.

The problem of synthetically generating IP traffic matrices: initial recommendations

There exist a wide variety of network design problems that require a traffic matrix as input in order to carry out performance evaluation. The research community has not had at its disposal any information about how to construct realistic traffic matrices. We introduce here the two basic problems that need to be addressed to construct such matrices. The first is that of synthetically generating traffic volume levels that obey spatial and temporal patterns as observed in realistic traffic matrices. The second is that of assigning a set of numbers (representing traffic levels) to particular node pairs in a given topology. This paper provides an in-depth discussion of the many issues that arise when addressing these problems. Our approach to the first problem is to extract statistical characteristics for such traffic from real data collected inside two large IP backbones. We dispel the myth that uniform distributions can be used to randomly generate numbers for populating a traffic matrix. Instead, we show that the lognormal distribution is better for this purpose as it describes well the mean rates of origin-destination flows. We provide estimates for the mean and variance properties of the traffic matrix flows from our datasets. We explain the second problem and discuss the notion of a traffic matrix being well-matched to a topology. We provide two initial solutions to this problem, one using an ILP formulation that incorporates simple and well formed constraints. Our second solution is a heuristic one that incorporates more challenging constraints coming from carrier practices used to design and evolve topologies.

Experiences in a 3G network: interplay between the wireless channel and applications

We present an experimental characterization of the physical and MAC layers in CDMA 1xEV-DO and their impact on transport layer performance. The 1xEV-DO network is currently the fastest mobile broadband cellular network, offering data rates of up to 3.1 Mbps for both stationary and mobile users. These rates are achieved by using novel capacity enhancement techniques at the lower layers. Specifically, 1xEV-DO incorporates rapid channel rate adaptation in response to signal conditions, and opportunistic scheduling to exploit channel fluctuations. Although shown to perform well in isolation, there is no comprehensive literature that examines the impact of these features on transport layer and application performance in real networks. We take the first step in addressing this issue through a large set of experiments conducted on a commercial 1xEV-DO network. Our evaluation includes both stationary and mobile scenarios wherein we transferred data using four popular transport protocols: TCPReno, TCP-Vegas, TCP-Westwood, and TCP-Cubic, and logged detailed measurements about wireless channel level characteristics as well as transport layer performance. We analyzed data from several days of experiments and inferred the properties of the physical, MAC and transport layers, as well as potential interactions between them. We find that the wireless channel data rate shows significant variability over long time scales on the order of hours, but retains high memory and predictability over small time scales on the order of milliseconds. We also find that loss-based TCP variants are largely unaffected by channel variations due to the presence of large buffers, and hence able to achieve in excess of 80% of the system capacity.

Blocking in all-optical networks

We present a new analytical technique, based on the inclusion-exclusion principle from combinatorial mathematics, for the analysis of all-optical networks with no wavelength conversion and random wavelength assignment. We use this technique to propose two models of low complexity for analysing networks with arbitrary topologies and traffic patterns. The first model improves the current technique by Birman (1996) in that the complexity of calculation is independent of hop-length and scales only with the capacity of the link as against that of Birmans method which grows exponentially with hop-length. We then propose a new heuristic to account for wavelength correlation and show that the second model is accurate even for sparse networks. Our technique can also be extended to analyse fixed alternate and least loaded routing.

EigenSpokes: surprising patterns and scalable community chipping in large graphs

We report a surprising, persistent pattern in large sparse social graphs, which we term EigenSpokes We focus on large Mobile Call graphs, spanning about 186K nodes and millions of calls, and find that the singular vectors of these graphs exhibit a striking EigenSpokes pattern wherein, when plotted against each other, they have clear, separate lines that often neatly align along specific axes (hence the term “spokes”) Furthermore, analysis of several other real-world datasets e.g. Patent Citations, Internet, etc. reveals similar phenomena indicating this to be a more fundamental attribute of large sparse graphs that is related to their community structure. This is the first contribution of this paper Additional ones include (a) study of the conditions that lead to such EigenSpokes, and (b) a fast algorithm for spotting and extracting tightly-knit communities, called SpokEn, that exploits our findings about the EigenSpokes pattern.

Impact of Packet Sampling on Portscan Detection

Packet sampling is commonly deployed in high-speed backbone routers to minimize resources used for network monitoring. It is known that packet sampling distorts traffic statistics and its impact has been extensively studied for traffic engineering metrics such as flow size and mean rate. However, it is unclear how packet sampling impacts anomaly detection, which has become increasingly critical to network providers. This paper is the first attempt to address this question by focusing on one common class of nonvolume-based anomalies, portscans, which are associated with worm/virus propagation. Existing portscan detection algorithms fall into two general approaches: target-specific and traffic profiling. We evaluated representative algorithms for each class, namely: 1) TRWSYN that performs stateful traffic analysis; 2) TAPS that tracks connection pattern of scanners; and 3) entropy-based traffic profiling. We applied these algorithms to detect portscans in both the original and sampled packet traces from a Tier-1 providers backbone network. Our results demonstrate that sampling introduces fundamental bias that degrades the effectiveness of these detection algorithms and dramatically increases false positives. Through both experiments and analysis, we identify the traffic features critical for anomaly detection that are affected by sampling. Finally, using insight gained from this study, we show how portscan algorithms can be enhanced to be more robust to sampling

Making IGP routing robust to link failures

An important requirement of a robust traffic engineering solution is insensitivity to changes, be they in the form of traffic fluctuations or changes in the network topology because of link failures. In this paper we focus on developing a fast and effective technique to compute traffic engineering solutions for Interior Gateway Protocol (IGPs) environments that are robust to link failures in the logical topology. The routing and packet forwarding decisions for IGPs is primarily governed by link weights. Our focus is on computing a single set of link weights for a traffic engineering instance that performs well over all single logical link failures. Such types of failures, although usually not long lasting, of the order of tens of minutes, can occur with high enough frequency, of the order of several a day, to significantly affect network performance. The relatively short duration of such failures coupled with issues of computational complexity and convergence time due to the size of current day networks discourage adaptive reactions to such events. Consequently, it is desirable to a priori compute a routing solution that performs well in all such scenarios. Through computational evaluations we demonstrate that our technique yields link weights that perform well over all single link failures and also scales well, in terms of computational complexity, with the size of the network.

Performance analysis of channelized cellular systems with dynamic channel allocation

We present an analytical model to compute the blocking probability in channelized cellular systems with dynamic channel allocation. We model the channel occupancy in a cell by a two-dimensional (2D) Markov chain, which can be solved to obtain the blocking probability in each cell. We apply our analytical model to linear highway systems with and without lognormal shadowing and then extend it to 2D cellular systems with lognormal shadowing. We show that, for linear highway systems, distributed dynamic channel-allocation schemes perform similarly to the centralized dynamic channel-allocation schemes in terms of blocking probability. However, for 2D cellular systems, the improvement in the performance is significant and the reduction in the blocking probability in systems with distributed dynamic channel allocation is by almost one order of magnitude, when compared to that in systems with centralized dynamic channel allocation. In practice, our analysis of linear highway systems is applicable to Digital European Cordless Telephony (DECT) and that of 2D cellular systems is applicable to global systems for mobile communications (GSM).

On the impact of aggregation on the performance of traffic aware routing

This paper investigates the impact of traffic aggregation on the performance of routing algorithms that incorporate traffic information. We focus on two issues. Firstly, we explore the relationship between average network performance and the coarseness (granularity) of traffic splitting across routes. Specifically, we are interested in how average network performance improves with our ability to distribute traffic arbitrarily across multiple paths. Secondly, we shift our attention from average to short-term performance, with again a focus on the impact of traffic granularity. In particular, we explore the relation between the level of traffic aggregation and its variability, which directly affects short-term routing performance. Our investigation relies on traffic traces collected from an operational network, and its results provide insight into the cost-performance trade-off associated with deploying “traffic aware” routing protocols.