Andreas Kind

Histogram-based traffic anomaly detection

Identifying network anomalies is essential in enterprise and provider networks for diagnosing events, like attacks or failures, that severely impact performance, security, and Service Level Agreements (SLAs). Feature-based anomaly detection models (ab)normal network traffic behavior by analyzing different packet header features, like IP addresses and port numbers. In this work, we describe a new approach to feature-based anomaly detection that constructs histograms of different traffic features, models histogram patterns, and identifies deviations from the created models. We assess the strengths and weaknesses of many design options, like the utility of different features, the construction of feature histograms, the modeling and clustering algorithms, and the detection of deviations. Compared to previous feature-based anomaly detection approaches, our work differs by constructing detailed histogram models, rather than using coarse entropy-based distribution approximations. We evaluate histogram-based anomaly detection and compare it to previous approaches using collected network traffic traces. Our results demonstrate the effectiveness of our technique in identifying a wide range of anomalies. The assessed technical details are generic and, therefore, we expect that the derived insights will be useful for similar future research efforts.

Probabilistic lossy counting: an efficient algorithm for finding heavy hitters

Knowledge of the largest traffic ows in a network is important for many network management applications. The problem of finding these ows is known as the heavy-hitter problem and has been the subject of many studies in the past years. One of the most efficient and well-known algorithms for finding heavy hitters is lossy counting [29]. In this work we introduce probabilistic lossy counting (PLC), which enhances lossy counting in computing network traffic heavy hitters. PLC uses on a tighter error bound on the estimated sizes of traffic ows and provides probabilistic rather than deterministic guarantees on its accuracy. The probabilistic-based error bound substantially improves the memory consumption of the algorithm. In addition, PLC reduces the rate of false positives of lossy counting and achieves a low estimation error, although slightly higher than that of lossy counting We compare PLC with state-of-the-art algorithms for finding heavy hitters. Our experiments using real traffic traces find that PLC has 1) between 34.4% and 74% lower memory consumption, 2) between 37.9% and 40.5% fewer false positives than lossy counting, and 3) a small estimation error.

On the 95-Percentile Billing Method

The 95-percentile method is used widely for billing ISPs and websites. In this work, we characterize important aspects of the 95-percentile method using a large set of traffic traces. We first study how the 95-percentile depends on the aggregation window size. We observe that the computed value often follows a noisy decreasing trend along a convex curve as the window size increases. We provide theoretical justification for this dependence using the self-similar model for Internet traffic and discuss observed more complex dependencies in which the 95-percentile increases with the window size. Secondly, we quantify how variations on the window size affect the computed 95-percentile. In our experiments, we find that reasonable differences in the window size can account for an increase between 4.1% and 42.5% in the monthly bill of medium and low-volume sites. In contrast, for sites with average traffic rates above 10Mbps the fluctuation of the 95-percentile is bellow 2.9%. Next, we focus on the use of flow data in hosting environments for billing individual sites. We describe the byte-shifting effect introduced by flow aggregation and quantify how it can affect the computed 95-percentile. We find that in our traces it can both decrease and increase the computed 95-percentile with the largest change being a decrease of 9.3%.

Advanced network monitoring brings life to the awareness plane

The latest advances in traffic measurement, analysis, and modeling play an important role in automatically building and maintaining a distributed intelligent monitoring layer that we describe as the awareness plane. The purpose of this article is to describe the components of this awareness plane in the areas of flexible network measurement, application and relationship discovery, and traffic classification, as well as data aggregation and semantically enriched infrastructure models. We present management services and scenarios, and list the research challenges on the path to integrating the components and making them interoperate for future autonomic service and network management approaches.

Creating advanced functions on network processors: experience and perspectives

In this article we present five case studies of advanced networking functions that detail how a network processor (NP) can provide high performance and also the necessary flexibility compared with ASIC. We first review the basic NP system architectures, and describe the IBM PowerNP architecture from the data plane as well as the control plane point of view. We introduce models for the programmers views of NP that facilitate a global understanding of NP software programming. Then, for each case study, we present results from prototypes as well as general considerations that apply to a wider range of system architectures. Specifically, we investigate the suitability of NP for QoS (active queue management and traffic engineering), header processing (GPRS tunneling protocol), intelligent forwarding (load balancing without flow disruption), payload processing (code interpretation and just-in-time compilation in active networks), and protocol stack termination (SCTP). Finally, we summarize the key features as revealed by each case study, and conclude with remarks on the future of NP.

A two-layered anomaly detection technique based on multi-modal flow behavior models

We present a novel technique to detect traffic anomalies based on network flow behavior in different traffic features. Based on the observation that a network has multiple behavior modes, we estimate the modes in each feature component and extract their model parameters during a learning phase. Observed network behavior is then compared to the baseline models by means of a two-layered distance computation: first, component-wise anomaly indices and second, a global anomaly index for each traffic feature enable effective detection of aberrant behavior. Our technique supports on-line detection and incorporation of administrator feedback and does not make use of explicit prior knowledge about normal and abnormal traffic. We expect benefits from the modeling and detection strategy chosen to reliably expose abnormal events of diverse nature at both detection layers while being resilient to seasonal effects. Experiments on simulated and real network traces confirm our expectations in detecting true anomalies without increasing the false positive rate. A comparison of our technique with entropy-and histogram-based approaches demonstrates its ability to reveal anomalies that disappear in the background noise of output signals from these techniques.

Relationship Discovery with NetFlow to Enable Business-Driven IT Management

The understanding of relationships and dependencies between business processes and the underlying IT infrastructure is important for enabling business-driven IT management. This work uses the widespread NetFlow feature to derive direct and indirect traffic relationships in IT infrastructures. We define an algorithm for relationship discovery with NetFlow and describe the application of the discovery approach in a large production environment.

Mining semantic relations using NetFlow

Knowing the dependencies among computing assets and services provides insights into the computing and business landscape, therefore, facilitating low-risk timely changes in support of a business-driven IT management. In general, the results of a dependency analysis can be used for infrastructure reengineering, show evidence of policy and process compliance, and support assessments of business resilience. Current passive discovery approaches using network monitoring analyze only direct communication between assets and provide just a single- link mesh view. This work introduces a new algorithm based on NetFlow data preprocessed by the Aurora system developed at IBM Research to create a dependency model of the network. The algorithm uses time-based event correlation and the data mining concept of association rules to detect and classify dependencies that span two or more components. The advantages of the algorithm is that no access credentials are required and no packet payload inspection is performed. The suggested algorithm populates and maintains a dependency model of an observed network that describes dependencies among computer systems, software components, and services. The model combines the mined association rules that express relations between flows into dependencies, which are given intuitive semantics. Tests with simulated and authentic data prove the accuracy of the dependency mining algorithm.

Bandwidth allocation for non-responsive flows with active queue management

This paper addresses the problem of configuring active queue management systems (e.g. WRED and RIO) for service level specifications in internetworks. In particular, we focus on assured forwarding (AF) for non-responsive flows in differentiated services networks. The difficulty is to determine the correct queue level thresholds that will result in correct drop rates for various AF precedence levels under any combination of offered loads. A new active queue management scheme based on a control algorithm is proposed that senses not only queue levels but also rates of queue levels changes and per flow bit rates to converge automatically to an optimal set of transmit fractions. The scheme has been implemented and tested on a network processor. Results show that the new active queue management scheme protects assured aggregated flow rates during periods of congestion. For non-responsive traffic the buffer occupancy level remains low during 250% offered load.

INCA: an agent-based network control architecture

This paper describes the design and implementation of INCA, an open architecture for the distributed management of multi-service networks and systems applications. The Intelligent Network Control Architecture is populated by stationary and mobile intelligent agents. These agents perform monitoring and control of network and systems components, thereby supporting the integrated management of networks and services. The architecture provides transaction capabilities to control transport and mobility of agents, agent prioritization, and multiple agent code transfer schemes. Managed objects used to access resources on network elements and new system functionality can be created, distributed, and replaced dynamically. An example INCA application demonstrates that prioritized agents are necessary to support the timely execution of critical tasks. The design of our agent based network management platform is not bound to a particular programming language or computing environment; the current implementation however, is based on Java and RMI.