Akihiro Shimoda

Spatio-temporal factorization of log data for understanding network events

Understanding the impacts and patterns of network events such as link flaps or hardware errors is crucial for diagnosing network anomalies. In large production networks, analyzing the log messages that record network events has become a challenging task due to the following two reasons. First, the log messages are composed of unstructured text messages generated by vendor-specific rules. Second, network equipment such as routers, switches, and RADIUS severs generate various log messages induced by network events that span across several geographical locations, network layers, protocols, and services. In this paper, we have tackled these obstacles by building two novel techniques: statistical template extraction (STE) and log tensor factorization (LTF). STE leverages a statistical clustering technique to automatically extract primary templates from unstructured log messages. LTF aims to build a statistical model that captures spatial-temporal patterns of log messages. Such spatial-temporal patterns provide useful insights into understanding the impacts and root cause of hidden network events. This paper first formulates our problem in a mathematical way. We then validate our techniques using massive amount of network log messages collected from a large operating network. We also demonstrate several case studies that validate the usefulness of our technique.

SFMap: Inferring Services over Encrypted Web Flows Using Dynamical Domain Name Graphs

Most modern Internet services are carried over the web. A significant amount of web transactions is now encrypted and the transition to encryption has made it difficult for network operators to understand traffic mix. The goal of this study is to enable network operators to infer hostnames within HTTPS traffic because hostname information is useful to understand the breakdown of encrypted web traffic. The proposed approach correlates HTTPS flows and DNS queries/responses. Although this approach may appear trivial, recent deployment and implementation of DNS ecosystems have made it a challenging research problem; i.e., canonical name tricks used by CDNs, the dynamic and diverse nature of DNS TTL settings, and incomplete measurements due to the existence of various caching mechanisms. To tackle these challenges, we introduce domain name graph (DNG), which is a formal expression that characterizes the highly dynamic and diverse nature of DNS mechanisms. Furthermore, we have developed a framework called Service-Flow map (SFMap) that works on top of the DNG. SFMap statistically estimates the hostname of an HTTPS server, given a pair of client and server IP addresses. We evaluate the performance of SFMap through extensive analysis using real packet traces collected from two locations with different scales. We demonstrate that SFMap establishes good estimation accuracies and outperforms a state-of-the-art approach.

Statistical estimation of the names of HTTPS servers with domain name graphs

We present the domain name graph (DNG), which is a formal expression that can keep track of CNAME chains and characterize the dynamic and diverse nature of DNS mechanisms and deployments.We develop a framework called Service-Flow map (SFMap) that works on top of the DNG. SFMap estimates the hostname of an HTTPS server when given a pair of client and server IP addresses. It can statistically estimate the hostname even when associating DNS queries are unobserved due to caching mechanisms, etc.Through extensive analysis using real packet traces, we demonstrate that the SFMap framework establishes good estimation accuracies and can out- perform the state-of-the art technique called DN-Hunter. We also identify the optimized setting of the SFMap framework. The experiment results suggest that the success of the SFMap lies in the fact that it can complement incomplete DNS information by leveraging the graph structure.To cope with large-scale measurement data, we introduce techniques to make the SFMap framework scalable. We validate the effectiveness of the approach using large-scale traffic data collected at a gateway point of Internet access links. Adoption of SSL/TLS to protect the privacy of web users has become increasingly common. In fact, as of September 2015, more than 68% of top-1M websites deploy SSL/TLS to encrypt their traffic. The transition from HTTP to HTTPS has brought a new challenge for network operators who need to understand the hostnames of encrypted web traffic for various reasons. To meet the challenge, this work develops a novel framework called SFMap, which estimates names of HTTPS servers by analyzing precedent DNS queries/responses in a statistical way. The SFMap framework introduces domain name graph, which can characterize highly dynamic and diverse nature of DNS mechanisms. Such complexity arises from the recent deployment and implementation of DNS ecosystems; i.e., canonical name tricks used by CDNs, the dynamic and diverse nature of DNS TTL settings, and incomplete and unpredictable measurements due to the existence of various DNS caching instances. First, we demonstrate that SFMap establishes good estimation accuracies and outperforms a state-of-the-art approach. We also aim to identify the optimized setting of the SFMap framework. Next, based on the preliminary analysis, we introduce techniques to make the SFMap framework scalable to large-scale traffic data. We validate the effectiveness of the approach using large-scale Internet traffic.

Sensor in the Dark: Building Untraceable Large-Scale Honeypots Using Virtualization Technologies

A Honeypot is a system that aims to detect and analyze malicious attacks attempted on a network in an interactive manner. Because the primary objective of a honeypot is to detect enemies without being known to them, it is important to hide its existence. However, as several studies have reported, exploiting the unique characteristics of hosts working on a consecutive IP addresses range easily reveals the existence of honeypots. In fact, there exist some anti-honeypot tools that intelligently probe IP address space to locate Internet security sensors including honeypots. In order to tackle this problem, we propose a system called DarkPots, that consists of a large number of virtualized honeypots using unused and nonconsecutive IP addresses in a production network. DarkPots enables us to deploy a large number of honeypots within an active IP space used for a production network; thus detection is difficult using existing probing techniques. In addition, by virtually classifying the unused IP addresses into several groups, DarkPots enables us to perform several monitoring schemes simultaneously. This function is meaningful because we can adopt more than one monitoring schemes and compare their results in an operating network. We design and implement a prototype of DarkPots and empirically evaluate its effectiveness and feasibility by concurrently performing three independent monitoring schemes in a high-speed campus network. The system successfully emulated 7,680 of virtualized honeypots on a backbone link that carries 500 Mbps – 1 Gbps of traffic without affecting legitimate traffic. Our key findings suggest: (1) active and interactive monitoring schemes provide more in-depth insights of malicious attacks, compared to passive monitoring approach in a quantitative way, and (2) randomly distributed allocation of IP addresses has an advantage over the concentrated allocation in that it can collect more information from malwares. These features are crucial in monitoring the security threats.

Inferring Popularity of Domain Names with DNS Traffic: Exploiting Cache Timeout Heuristics

Popularity ranking of Internet services is an important metric for network operators, because it enables mid- to-long term planning of their network facilities and root cause analysis for unexpected traffic. The service-oriented traffic monitoring is much helpful to infer the popularity, hence it has been gathering much attention from both researchers and practitioners. Lately, service identification of a given flow has become very difficult due to the rapid growth of CDNs and/or encrypted traffic, while some research works employed preceding DNS traffic as a hint. However, because of its cache mechanism, the DNS message count deviates from the actual number of flows, which can greatly degrade the ranking reliability. We propose a theoretical model for inferring the users number of accesses per domain name by exploiting the characteristics of the DNS message count. To the best of our knowledge, this paper is the first attempt to formulate the effect of users stub resolvers; previous studies were focused on analyzing the effect of cache servers. We evaluated the precision of our model with a real dataset of traffic of thousands of users. By analyzing the top-50 domain names by the number of users, we can infer the number of flows within a 24% error rate on average in 42 out of 50 FQDNs.

i-Path: Improving Path Visibility for the Future Internet

In this paper, we present the concept design, and implementation of a novel network measurement system for the future Internet. The new protocol offers end-point applications a mechanism for utilizing internal information to maximize transport. By a cross-layer approach, we can automatically to collect information along a path while upholding a disclosure policy for the information. The protocol has been implemented on commonly used operating systems and has been tested on both commercial and test-bed networks. A peer-to-peer file sharing application has been modified to support the protocol and experiments shows that download times were reduced and bandwidth was used more efficiently.