Javier Ramos

High-Performance network traffic processing systems using commodity hardware

The Internet has opened new avenues for information accessing and sharing in a variety of media formats. Such popularity has resulted in an increase of the amount of resources consumed in backbone links, whose capacities have witnessed numerous upgrades to cope with the ever-increasing demand for bandwidth. Consequently, network traffic processing at todays data transmission rates is a very demanding task, which has been traditionally accomplished by means of specialized hardware tailored to specific tasks. However, such approaches lack either of flexibility or extensibility--or both. As an alternative, the research community has pointed to the utilization of commodity hardware, which may provide flexible and extensible cost-aware solutions, ergo entailing large reductions of the operational and capital expenditure investments. In this chapter, we provide a survey-like introduction to high-performance network traffic processing using commodity hardware. We present the required background to understand the different solutions proposed in the literature to achieve high-speed lossless packet capture, which are reviewed and compared.

Multi-granular, multi-purpose and multi-Gb/s monitoring on off-the-shelf systems

SUMMARY As an attempt to make network managers’ life easier, we present M3Omon, a system architecture that helps to develop monitoring applications and perform network diagnosis. M3Omon behaves as an intermediate layer between the traffic and monitoring applications that provides advanced features, high performance and low cost. Such advanced features leverage a multi-granular and multi-purpose approach to the monitoring problem. Multi-granular monitoring provides answers to tasks that use traffic aggregates to identify an event, and requires either flow records or packet data or even both to understand it and, eventually, take convenient countermeasures. M3Omon provides a simple API to access traffic simultaneously at several different granularities, i.e. packet-level, flow-level and aggregate statistics. The multi-purposed design of M3Omon allows not only performing tasks in parallel that are specifically targeted to different traffic-related purposes (e.g. traffic classification and intrusion detection) but also sharing granularities between applications, e.g. several concurrent applications fed from flow records that are provided by M3Omon. Finally, the low-cost characteristic is brought by off-the-shelf systems (the combination of open-source software and commodity hardware) and the high performance is achieved thanks to modifications in the standard NIC driver, low-level hardware interaction, efficient memory management and programming optimization. Copyright

Batch to the Future: Analyzing Timestamp Accuracy of High-Performance Packet I/O Engines

Novel packet I/O engines allow capturing traffic at multi-10Gb/s using only-software and commodity-hardware systems. This is achieved thanks to the application of techniques such as batch processing. Nevertheless, this feature involves degradation in the timestamp accuracy, which may be relevant for monitoring purposes. We propose two different approaches to mitigate such effect: a simple algorithm to distribute inter-batch time among the packets composing a batch, and a driver modification to poll NIC buffers avoiding batch processing. Experimental results, using both synthetic and real traffic, show that our proposals allow capturing accurately timestamped traffic for monitoring purposes at multi-10Gb/s rates.

Multi‐granular, multi‐purpose and multi‐Gb/s monitoring on off‐the‐shelf systems

SUMMARY As an attempt to make network managers’ life easier, we present M3Omon, a system architecture that helps to develop monitoring applications and perform network diagnosis. M3Omon behaves as an intermediate layer between the traffic and monitoring applications that provides advanced features, high performance and low cost. Such advanced features leverage a multi-granular and multi-purpose approach to the monitoring problem. Multi-granular monitoring provides answers to tasks that use traffic aggregates to identify an event, and requires either flow records or packet data or even both to understand it and, eventually, take convenient countermeasures. M3Omon provides a simple API to access traffic simultaneously at several different granularities, i.e. packet-level, flow-level and aggregate statistics. The multi-purposed design of M3Omon allows not only performing tasks in parallel that are specifically targeted to different traffic-related purposes (e.g. traffic classification and intrusion detection) but also sharing granularities between applications, e.g. several concurrent applications fed from flow records that are provided by M3Omon. Finally, the low-cost characteristic is brought by off-the-shelf systems (the combination of open-source software and commodity hardware) and the high performance is achieved thanks to modifications in the standard NIC driver, low-level hardware interaction, efficient memory management and programming optimization. Copyright

On the processing time for detection of Skype traffic

The last few years have witnessed VoIP applications gaining a tremendous popularity and Skype, in particular, is leading this continuous expansion. Unfortunately, Skype follows a closed source and proprietary design, and typically uses encryption mechanisms, making it very difficult to identify its presence from a traffic aggregate. Several algorithms and approaches have been proposed to perform such task with promising results in terms of accuracy. However, such approaches typically require significant computation resources and it is unlikely that they can be deployed in nowadays high-speed networks. In this light, this paper focuses on cutting the processing cost of algorithms to detect Skype traffic. We have conveniently tuned a previous well-validated algorithm and we have assessed its performance. To this end, we have used real traces from public repositories, from a Spanish 3G operator, and synthetic traces. Our results show that a single process can detect Skype traffic at 1 Gbps rates reading replayed real traces directly from a NIC. Even more, 3.7 Gbps are achieved reading from traces previously allocated in memory using a single process and 45 Gbps using 16 concurrent processes. This fact paves the way for 10 Gbps processing in commodity hardware.

ETOMIC Advanced Network Monitoring System for Future Internet Experimentation

ETOMIC is a network traffic measurement platform with high precision GPS-synchronized monitoring nodes. The infrastructure is publicly available to the network research community, supporting advanced experimental techniques by providing high precision hardware equipments and a Central Management System. Researchers can deploy their own active measurement codes to perform experiments on the public Internet. Recently, the functionalities of the original system were significantly extended and new generation measurement nodes were deployed. The system now also includes well structured data repositories to archive and share raw and evaluated data. These features make ETOMIC as one of the experimental facilities that support the design, development and validation of novel experimental techniques for the future Internet. In this paper we focus on the improved capabilities of the management system, the recent extensions of the node architecture and the accompanying database solutions.

Testing the capacity of off-the-shelf systems to store 10GbE traffic

The maturity of the telecommunications market and the fact that user demands increase every day leaves network operators no option but to deploy high-speed infrastructures and test them in an efficient and economical manner. A common approach to this problem has been the storage of network traffic samples for analysis and replay using different versions of what we have named NTSS. This type of task is particularly demanding in 10 Gb Ethernet links and has traditionally been addressed by closed solutions or NTSS built on top of high-end hardware. However, these approaches lack flexibility and extensibility, which typically translates into higher cost. This work studies how NTSS can be built using COTS: a combination of commodity hardware and open source software. To this end, we present the current limitations of COTS systems and focus on low-level optimization techniques at several levels: the NIC driver, hard drives, and the software interaction between them. The application of these techniques has proven crucial for reaching 10 Gb/s rates, as different state-of-the-art systems have shown after an extensive performance test.

Packet Storage at Multi-gigabit Rates Using Off-the-Shelf Systems

The use of closed solutions from most known vendors to carry out network-monitoring tasks has turned out to be a questionable option given their lack of flexibility and extensibility, which has typically been translated into higher costs. Consequently, we study whether high-performance monitoring tasks can be carried out using off-the-shelf systems, the alternative to these pitfalls from the research community, consisting in the combination of open-source software and commodity hardware. We focus on sniffing and storing network traffic as one of the major tasks in any monitoring architecture. Specifically, we first review the keys to sniff traffic at multi-gigabit rates, and then present an experimental evaluation of commodity hard drives. Finally, the lessons learned from such studies and the performed experiments have conducted us to the development of an open solution, namely HPCAP, which sniffs and stores multi-gigabit traffic using commodity hardware without packet losses in very demanding scenarios.

On the effect of concurrent applications in bandwidth measurement speedometers

Nowadays, operators are facing an increasing demand to ensure the committed bandwidth in broadband access links, both from customers and regulatory bodies. Usually, customers employ speedometers (such as Speedtest) to evaluate the access link performance. Such speedometers run in the user’s PC, concurrently with many other applications which may generate cross-traffic and reduce the processing time to the remaining applications. In this paper we analyze the impact of cross-traffic from other applications and CPU and memory occupancy due to concurrent applications in the speedometer accuracy, for a broad range of file-download and packet-train-based measurement techniques. We conclude that the packet-train technique is highly immune to cross-traffic, CPU and memory occupancy, which makes it amenable to use in general-purpose PCs running concurrent software. However, an estimation of CPU, memory occupancy and cross-traffic from other applications improves the measurement accuracy. In this regard, we present the QoS-Poll measurement software, which estimates the occupancy of system resources, including cross-traffic from other applications, and defines a region of acceptance of the bandwidth measurement in terms of the previous system parameters. The results of a QoS-Poll measurement campaign (more than 75 households) are also reported, showing excellent measurement accuracy.

Accurate and affordable packet-train testing systems for multi-gigabit-per-second networks

Communication networks these days face a relentless increase in traffic load. Multi-gigabitper- second links are becoming widespread, and network devices are under continuous stress, so testing whether they guarantee the specified throughput or delay is a must. Software-based solutions, such as packet-train traffic injection, were adequate for lower speeds, but they have become inaccurate in the current scenario. Hardware-based solutions have proved to be very accurate, but usually at the expense of much higher development and acquisition costs. Fortunately, new affordable FPGA SoC devices, as well as high-level synthesis tools, can very efficiently reduce these costs. In this article we show the advantages of hardware-based solutions in terms of accuracy, comparing the results obtained in an FPGA SoC development platform and in NetFPGA-10G to those of software. Results show that a hardware-based solution is significantly better, especially at 10 Gb/s. By leveraging high-level synthesis and open source platforms, prototypes were quickly developed. Noticeable advantages of our proposal are high accuracy, competitive cost with respect to the software counterpart, which runs in high-end off-the-shelf workstations, and the capability to easily evolve to upcoming 40 Gb/s and 100 Gb/s networks.