Mehdi Malboubi

Intelligent SDN based traffic (de)Aggregation and Measurement Paradigm (iSTAMP)

Fine-grained traffic flow measurement, which provides useful information for network management tasks and security analysis, can be challenging to obtain due to monitoring resource constraints. The alternate approach of inferring flow statistics from partial measurement data has to be robust against dynamic temporal/spatial fluctuations of network traffic. In this paper, we propose an intelligent Traffic (de)Aggregation and Measurement Paradigm (iSTAMP), which partitions TCAM entries of switches/routers into two parts to: 1) optimally aggregate part of incoming flows for aggregate measurements, and 2) de-aggregate and directly measure the most informative flows for per-flow measurements. iSTAMP then processes these aggregate and per-flow measurements to effectively estimate network flows using a variety of optimization techniques. With the advent of Software-Defined-Networking (SDN), such real-time rule (re)configuration can be achieved via OpenFlow or other similar SDN APIs. We first show how to design the optimal aggregation matrix for minimizing the flow-size estimation error. Moreover, we propose a method for designing an efficient-compressive flow aggregation matrix under hard resource constraints of limited TCAM sizes. In addition, we propose an intelligent Multi-Armed Bandit based algorithm to adaptively sample the most “rewarding” flows, whose accurate measurements have the highest impact on the overall flow measurement and estimation performance. We evaluate the performance of iSTAMP using real traffic traces from a variety of network environments and by considering two applications: traffic matrix estimation and heavy hitter detection. Also, we have implemented a prototype of iSTAMP and demonstrated its feasibility and effectiveness in Mininet environment.

Decentralizing Network Inference Problems With Multiple-Description Fusion Estimation MDFE

Two forms of network inference (or tomography) problems have been studied rigorously: (a) traffic matrix estimation or completion based on link-level traffic measurements, and (b) link-level loss or delay inference based on end-to-end measurements. These problems are often posed as underdetermined linear inverse (UDLI) problems and solved in a centralized manner, where all the measurements are collected at a central node, which then applies a variety of inference techniques to estimate the attributes of interest. This paper proposes a novel framework for decentralizing these large-scale UDLI network inference problems by intelligently partitioning it into smaller sub-problems and solving them independently and in parallel. The resulting estimates, referred to as multiple descriptions, can then be fused together to compute the global estimate. We apply this Multiple Description and Fusion Estimation (MDFE) framework to three classical problems: traffic matrix estimation, traffic matrix completion, and loss inference. Using real topologies and traces, we demonstrate how MDFE can speed up computation time while maintaining (even improving) the estimation accuracy and how it enhances robustness against noise and failures. We also show that our MDFE framework is compatible with a variety of existing inference techniques used to solve the UDLI problems.

OpenMeasure: Adaptive flow measurement & inference with online learning in SDN.

Accurate and efficient network-wide traffic measurement is crucial for network management. Recently, Software-defined networking (SDN) has opened up new opportunities in network measurement and inference. In this work, we demonstrate an efficient flow measurement and inference framework which performs adaptive measurement with online learning. Using the reprogrammability of SDN, we assist network inference with online learning predictions and dynamically update the measurement rules network-wide to track and measure the most informative flows. To best utilize the available measurement resources, we leverage the SDN controller (with its global view) to optimally place flow monitoring rules across network switches. Using real-world data, we show that our measurement framework achieves high performance in both estimating the traffic matrix and identifying hierarchical heavy hitters.

Software Defined Network Inference with Passive/Active Evolutionary-Optimal pRobing (SNIPER)

A key requirement for network management is the accurate and reliable monitoring of relevant network characteristics. In todays large-scale networks, this is a challenging task due to the hard constraints of network measurement resources. This paper proposes a new framework, SNIPER, which leverages the flexibility provided by Software-Defined Networking (SDN) to design the optimal observation or measurement matrix that can leads to the best achievable estimation accuracy using Matrix Completion (MC) techniques. To cope with the complexity of designing large-scale optimal observation matrices, we use the Evolutionary Optimization Algorithms (EOA) which directly target the ultimate estimation accuracy as the optimization objective function. We evaluate the performance of SNIPER using both synthetic and real network measurement traces from different network topologies and by considering two main applications including per-flow size and delay estimations. Our results show that SNIPER can be applied to a variety of network performance measurements under hard resource constraints. For example, by measuring 8.8\% of per-flow path delays in Harvard network, congested paths can be detected with probability 0.94. To demonstrate the feasibility of our framework, we also have implemented a prototype of SNIPER in Mininet.

Compressive sensing network inference with multiple-description fusion estimation

We have previously introduced Multiple Description Fusion Estimation (MDFE) framework that partitions a large-scale Under-Determined Linear Inverse (UDLI) problem into smaller sub-problems that can be solved independently and in parallel. The resulting estimates, referred to as multiple descriptions, can then be fused together to compute the global estimate. In this paper, we extend MDFE framework to make it compatible with Compressive Sensing (CS) network inference, where the attributes of interests (i.e. unknowns) are fluctuating rapidly over time and/or space. For this purpose, we propose a new clustering based technique to intelligently divide a large-scale compressive sensing problem into smaller sub-problems where observations between sub-spaces contain redundancy. We apply this new framework, referred to as Compressive Sensing MDFE (CS-MDFE), to three classical inference problems in networking: traffic matrix estimation, traffic matrix completion, and loss inference. Using real topologies and traces, we demonstrate how CS-MDFE can improve the estimation accuracy and speed up computation time, and how it enhances robustness against noise and failures. We also show that this framework is compatible with different CS inference techniques.

Decentralizing network inference problems with Multiple-Description Fusion Estimation (MDFE)

Network inference (or tomography) problems, such as traffic matrix estimation or completion and link loss inference, have been studied rigorously in different networking applications. These problems are often posed as under-determined linear inverse (UDLI) problems and solved in a centralized manner, where all the measurements are collected at a central node, which then applies a variety of inference techniques to estimate the attributes of interest. This paper proposes a novel framework for decentralizing these large-scale under-determined network inference problems by intelligently partitioning it into smaller subproblems and solving them independently and in parallel. The resulting estimates, referred to as multiple descriptions, can then be fused together to compute the global estimate. We apply this Multiple Description and Fusion Estimation (MDFE) framework to three classical problems: traffic matrix estimation, traffic matrix completion, and loss inference. Using real topologies and traces, we demonstrate how MDFE can speed up computation while maintaining (even improving) the estimation accuracy and how it enhances robustness against noise and failures. We also show that our MDFE framework is compatible with a variety of existing inference techniques used to solve the UDLI problems.

Parallelizing Under-Determined Inverse Problems for Network Applications

In this paper, we introduce a new technique for partitioning a large-scale under-determined linear inverse problem into multiple smaller subproblems that can be efficiently solved independently, and in parallel. When it is impossible or inefficient to solve a large-scale under-determined linear inverse problem, this technique can be used to significantly speed up the computation process without compromising the accuracy of the solution. We present numerical results that show the effectiveness of this approach when applied to network inference problems including traffic matrix estimation and network anomaly detection, both are important for managing large, complex networks and cyber security. Our proposed framework is applicable to other emerging applications in computational intelligence that can be formulated as under-determined linear inverse problems.

A learning-based measurement framework for traffic matrix inference in software defined networks

Abstract In this paper, we propose an intelligent framework for Traffic Matrix (TM) inference in Software Defined Networks (SDN) where the Ternary Content Addressable Memory (TCAM) entries of switches are partitioned into two parts to: 1) effectively aggregate part of incoming flows for aggregate measurements, and 2) de-aggregate and directly measure the most informative flows for per-flow measurements. These measurements are then processed to effectively estimate the size of network flows. Under hard resource constraints of limited TCAM sizes, we show how to design the optimal and efficient-compressed flow aggregation matrices. We propose an optimal Multi-Armed Bandit (MAB) based algorithm to adaptively measure the most rewarding flows. We evaluate the performance of our framework using real traffic traces from different network environments and by considering two main applications: TM estimation and Heavy Hitter (HH) detection. Moreover, we have implemented a prototype of our framework in Mininet to demonstrate its effectiveness.

Software defined network inference with evolutionary optimal observation matrices

Abstract A key requirement for network management is the accurate and reliable monitoring of relevant network characteristics. In today’s large-scale networks, this is a challenging task due to the scarcity of network measurement resources and the hard constraints that this imposes. This paper proposes a new framework, called SNIPER, which leverages the flexibility provided by Software-Defined Networking (SDN) to design the optimal observation or measurement matrix that can lead to the best achievable estimation accuracy using Matrix Completion (MC) techniques. To cope with the complexity of designing large-scale optimal observation matrices, we use the Evolutionary Optimization Algorithms (EOA) which directly target the ultimate estimation accuracy as the optimization objective function. We evaluate the performance of SNIPER using both synthetic and real network measurement traces from different network topologies and by considering two main applications for per-flow size and delay estimations. Our results show that SNIPER can be applied to a variety of network performance measurements under hard resource constraints. For example, by measuring only 8.8% of all per-flow path delays in Harvard network [1], congested paths can be detected with probability of 0.94. To demonstrate the feasibility of our framework, we also have implemented a prototype of SNIPER in Mininet.

Practical Approach to Identifying Additive Link Metrics with Shortest Path Routing

We revisit the problem of identifying link metrics from end- to-end path measurements in practical IP networks where shortest path routing is the norm. Previous solutions rely on explicit routing techniques (e.g., source routing or MPLS) to construct independent measurement paths for efficient link metric identification. However, most IP networks still adopt shortest path routing paradigm, while the explicit routing is not supported by most of the routers. Thus, this paper studies the link metric identification problem under shortest path routing constraints. To uniquely identify the link metrics, we need to place sufficient number of monitors into the network such that there exist