Rick Hofstede

Flow monitoring explained: From packet capture to data analysis with NetFlow and IPFIX

Flow monitoring has become a prevalent method for monitoring traffic in high-speed networks. By focusing on the analysis of flows, rather than individual packets, it is often said to be more scalable than traditional packet-based traffic analysis. Flow monitoring embraces the complete chain of packet observation, flow export using protocols such as NetFlow and IPFIX, data collection, and data analysis. In contrast to what is often assumed, all stages of flow monitoring are closely intertwined. Each of these stages therefore has to be thoroughly understood, before being able to perform sound flow measurements. Otherwise, flow data artifacts and data loss can be the consequence, potentially without being observed. This paper is the first of its kind to provide an integrated tutorial on all stages of a flow monitoring setup. As shown throughout this paper, flow monitoring has evolved from the early 1990s into a powerful tool, and additional functionality will certainly be added in the future. We show, for example, how the previously opposing approaches of deep packet inspection and flow monitoring have been united into novel monitoring approaches.

Booters — An analysis of DDoS-as-a-service attacks

In 2012, the Dutch National Research and Education Network, SURFnet, observed a multitude of Distributed Denial of Service (DDoS) attacks against educational institutions. These attacks were effective enough to cause the online exams of hundreds of students to be cancelled. Surprisingly, these attacks were purchased by students from Web sites, known as Booters. These sites provide DDoS attacks as a paid service (DDoS-as-a-Service) at costs starting from 1 USD. Since this problem was first identified by SURFnet, Booters have been used repeatedly to perform attacks on schools in SURFnets constituency. Very little is known, however, about the characteristics of Booters, and particularly how their attacks are structure. This is vital information needed to mitigate these attacks. In this paper we analyse the characteristics of 14 distinct Booters based on more than 250 GB of network data from real attacks. Our findings show that Booters pose a real threat that should not be underestimated, especially since our analysis suggests that they can easily increase their firepower based on their current infrastructure.

SSHCure: a flow-based SSH intrusion detection system

SSH attacks are a main area of concern for network managers, due to the danger associated with a successful compromise. Detecting these attacks, and possibly compromised victims, is therefore a crucial activity. Most existing network intrusion detection systems designed for this purpose rely on the inspection of individual packets and, hence, do not scale to todays high-speed networks. To overcome this issue, this paper proposes SSHCure, a flow-based intrusion detection system for SSH attacks. It employs an efficient algorithm for the real-time detection of ongoing attacks and allows identification of compromised attack targets. A prototype implementation of the algorithm, including a graphical user interface, is implemented as a plugin for the popular NfSen monitoring tool. Finally, the detection performance of the system is validated with empirical traffic data.

SSH Compromise Detection using NetFlow/IPFIX

Flow-based approaches for SSH intrusion detection have been developed to overcome the scalability issues of host-based alternatives. Although the detection of many SSH attacks in a flow-based fashion is fairly straightforward, no insight is typically provided in whether an attack was successful. We address this shortcoming by presenting a detection algorithm for the flow-based detection of compromises, i.e., hosts that have been compromised during an attack. Our algorithm has been implemented as part of our open-source IDS SSHCure and validated using almost 100 servers, workstations and honeypots, featuring an accuracy close to 100%.

SURFmap: A network monitoring tool based on the Google Maps API

Network monitoring allows network managers to get a better insight in the network traffic transiting in a managed network. In order to make the tasks of a network manager easier, many network monitoring tools are made available for a wide range of purposes (e.g., traffic accounting, performance analysis, and so on) network managers may have. However, most of these tools lack to provide geographical information about network traffic. This paper presents a network monitoring tool prototype, called SURFmap, which provides network traffic information at a geographical dimension by using the Google Maps API. Through the use of the Google Maps APIs features, SURFmap provides different zoom levels when showing network information, which results in the creation of different levels of abstraction in the network data visualization. SURFmap has revealed to be more intuitive when showing network traffic information, which makes the network monitoring activity from the network managers perspective more interesting.

Towards real-time intrusion detection for NetFlow and IPFIX

DDoS attacks bring serious economic and technical damage to networks and enterprises. Timely detection and mitigation are therefore of great importance. However, when flow monitoring systems are used for intrusion detection, as it is often the case in campus, enterprise and backbone networks, timely data analysis is constrained by the architecture of NetFlow and IPFIX. In their current architecture, the analysis is performed after certain timeouts, which generally delays the intrusion detection for several minutes. This paper presents a functional extension for both NetFlow and IPFIX flow exporters, to allow for timely intrusion detection and mitigation of large flooding attacks. The contribution of this paper is threefold. First, we integrate a lightweight intrusion detection module into a flow exporter, which moves detection closer to the traffic observation point. Second, our approach mitigates attacks in near real-time by instructing firewalls to filter malicious traffic. Third, we filter flow data of malicious traffic to prevent flow collectors from overload. We validate our approach by means of a prototype that has been deployed on a backbone link of the Czech national research and education network CESNET.

The network data handling war: MySQL vs. NfDump

Network monitoring plays a crucial role in any network management environment. Especially nowadays, with network speed and load constantly increasing, more and more data needs to be collected and efficiently processed. In highly interactive network monitoring systems, a quick response time from information sources turns out to be a crucial requirement. However, for data sets in the order of several GBs, this goal becomes difficult to achieve. In this paper, we present our operational experience in dealing with large amounts of network data. In particular, we focus on MySQL and NfDump, testing their capabilities under different usage scenarios and increasing data set sizes.

Towards multi-layered intrusion detection in high-speed networks

Traditional Intrusion Detection approaches rely on the inspection of individual packets, often referred to as Deep Packet Inspection (DPI), where individual packets are scanned for suspicious patterns. However, the rapid increase of link speeds and throughputs - especially in larger networks such as backbone networks - seriously constrains this approach. First, devices capable of detecting intrusions on high-speed links of 10 Gbps and higher are rather expensive, or must be built based on complex arrays. Second, legislation commonly restricts the way in which backbone network operators can analyse the data in their networks. To overcome these constraints, flow-based intrusion detection can be applied, which traditionally focuses only on packet header fields and packet characteristics. Flow export technologies are nowadays embedded in most high-end packet forwarding devices and are widely used for network management, which makes this approach economically attractive.

Real-time DDoS attack detection for Cisco IOS using NetFlow

Flow-based DDoS attack detection is typically performed by analysis applications that are installed on or close to a flow collector. Although this approach allows for easy deployment, it makes detection far from real-time and susceptible to DDoS attacks for the following reasons. First, the fact that the flow export process is timeout-based and that flow collectors typically provide data to analysis applications in chunks, can result in detection delays in the order of several minutes. Second, by the nature of flow export, attack traffic may be amplified by the flow export process if the original packets are small enough and are part of small flows. We have shown in a previous work how to perform DDoS attack detection on a flow exporter instead of a flow collector, i.e., close to the data source and in a real-time fashion, which however required access to a fully-extendible flow monitoring infrastructure. In this work, we investigate whether it is possible to operate the same detection system on a widely deployed networking platform: Cisco IOS. Since our ultimate goal is to identify besides the presence of an attack also attackers and targets, we rely on NetFlow. In this context, we present our DDoS attack detection prototype that has shown to generate a constant load on the underlying platform - even under attacks - underlining that DDoS attack detection can be performed on a Cisco Catalyst 6500 in production networks, if enough spare capacity is available.

Flow monitoring experiences at the ethernet-layer

Flow monitoring is a scalable technology for providing summaries of network activity. Being deployed at the IP-layer, it uses fixed flow definitions, based on fields of the IP-layer and higher layers. Since several backbone network operators are considering the deployment of (Carrier) Ethernet in their Next-Generation Network, flow monitoring should also evolve in that direction. In order to do flow monitoring at the Ethernet-layer, Ethernet header fields need to be considered in flow definitions. IPFIX provides the flexibility to change the definition of flows, incorporating information from several layers in the network (including non-IP fields). The deployment of IPFIX is still at an early stage, which means that use cases for Ethernet-layer monitoring are not well known yet. This paper provides an overview of the usability of IPFIX at the Ethernet-layer and presents several use cases in which Ethernet-layer flow monitoring provides new insights and different views on a network.