David Dagon

PolyUnpack: Automating the Hidden-Code Extraction of Unpack-Executing Malware

Modern malware often hide the malicious portion of their program code by making it appear as data at compile-time and transforming it back into executable code at runtime. This obfuscation technique poses obstacles to researchers who want to understand the malicious behavior of new or unknown malware and to practitioners who want to create models of detection and methods of recovery. In this paper we propose a technique for automating the process of extracting the hidden-code bodies of this class of malware. Our approach is based on the observation that sequences of packed or hidden code in a malware instance can be made self-identifying when its runtime execution is checked against its static code model. In deriving our technique, we formally define the unpack-executing behavior that such malware exhibits and devise an algorithm for identifying and extracting its hidden-code. We also provide details of the implementation and evaluation of our extraction technique; the results from our experiments on several thousand malware binaries show our approach can be used to significantly reduce the time required to analyze such malware, and to improve the performance of malware detection tools.

A Taxonomy of Botnet Structures

We propose a taxonomy of botnet structures, based on their utility to the botmaster. We propose key metrics to measure their utility for various activities (e.g., spam, ddos). Using these performance metrics, we consider the ability of different response techniques to degrade or disrupt botnets. In particular, our models show that targeted responses are particularly effective against scale free botnets and efforts to increase the robustness of scale free networks comes at a cost of diminished transitivity. Botmasters do not appear to have any structural solutions to this problem in scale free networks. We also show that random graph botnets (e.g., those using P2P formations) are highly resistant to both random and targeted responses. We evaluate the impact of responses on different topologies using simulation and demonstrate the utility of our proposed metrics by performing novel measurements of a P2P network. Our analysis shows how botnets may be classified according to structure and given rank or priority using our proposed metrics. This may help direct responses and suggests which general remediation strategies are more likely to succeed.

HoneyStat: Local Worm Detection Using Honeypots

Worm detection systems have traditionally used global strategies and focused on scan rates. The noise associated with this approach requires statistical techniques and large data sets (e.g., 220 monitored machines) to yield timely alerts and avoid false positives. Worm detection techniques for smaller local networks have not been fully explored.

Detecting Malicious Flux Service Networks through Passive Analysis of Recursive DNS Traces

In this paper we propose a novel, passive approach for detecting and tracking malicious flux service networks. Our detection system is based on passive analysis of recursive DNS (RDNS) traffic traces collected from multiple large networks. Contrary to previous work, our approach is not limited to the analysis of suspicious domain names extracted from spam emails or precompiled domain blacklists. Instead, our approach is able to detect malicious flux service networks in-the-wild, i.e., as they are accessed by users who fall victims of malicious content advertised through blog spam, instant messaging spam, social website spam, etc., beside email spam. We experiment with the RDNS traffic passively collected at two large ISP networks. Overall, our sensors monitored more than 2.5 billion DNS queries per day from millions of distinct source IPs for a period of 45 days. Our experimental results show that the proposed approach is able to accurately detect malicious flux service networks. Furthermore, we show how our passive detection and tracking of malicious flux service networks may benefit spam filtering applications.

Worm detection, early warning and response based on local victim information

Worm detection systems have traditionally focused on global strategies. In the absence of a global worm detection system, we examine the effectiveness of local worm detection and response strategies. This paper makes three contributions: (1) we propose a simple two-phase local worm victim detection algorithm, DSC (Destination-Source Correlation), based on worm behavior in terms of both infection pattern and scanning pattern. DSC can detect zero-day scanning worms with a high detection rate and very low false positive rate. (2) We demonstrate the effectiveness of early worm warning based on local victim information. For example, warning occurs with 0.19% infection of all vulnerable hosts on Internet when using a /12 monitored network. (3) Based on local victim information, we investigate and evaluate the effectiveness of an automatic real-time local response in terms of slowing down the global Internet worms propagation. (2) and (3) are general results, not specific to certain detection algorithm like DSC. We demonstrate (2) and (3) with both analytical models and packet-level network simulator experiments.

Measuring intrusion detection capability: an information-theoretic approach

A fundamental problem in intrusion detection is what metric(s) can be used to objectively evaluate an intrusion detection system (IDS) in terms of its ability to correctly classify events as normal or intrusive. Traditional metrics (e.g., true positive rate and false positive rate) measure different aspects, but no single metric seems sufficient to measure the capability of intrusion detection systems. The lack of a single unified metric makes it difficult to fine-tune and evaluate an IDS. In this paper, we provide an in-depth analysis of existing metrics. Specifically, we analyze a typical cost-based scheme [6], and demonstrate that this approach is very confusing and ineffective when the cost factor is not carefully selected. In addition, we provide a novel information-theoretic analysis of IDS and propose a new metric that highly complements cost-based analysis. When examining the intrusion detection process from an information-theoretic point of view, intuitively, we should have less uncertainty about the input (event data) given the IDS output (alarm data). Thus, our new metric, CI D (Intrusion Detection Capability), is defined as the ratio of the mutual information between the IDS input and output to the entropy of the input. CI D has the desired property that: (1) It takes into account all the important aspects of detection capability naturally, i.e., true positive rate, false positive rate, positive predictive value, negative predictive value, and base rate; (2) it objectively provides an intrinsic measure of intrusion detection capability; and (3) it is sensitive to IDS operation parameters such as true positive rate and false positive rate, which can demonstrate the effect of the subtle changes of intrusion detection systems. We propose CI D as an appropriate performance measure to maximize when fine-tuning an IDS. The obtained operation point is the best that can be achieved by the IDS in terms of its intrinsic ability to classify input data. We use numerical examples as well as experiments of actual IDSs on various data sets to show that by using CI D, we can choose the best (optimal) operating point for an IDS and objectively compare different IDSs.

Increased DNS forgery resistance through 0x20-bit encoding: security via leet queries

We describe a novel, practical and simple technique to make DNS queries more resistant to poisoning attacks: mix the upper and lower case spelling of the domain name in the query. Fortuitously, almost all DNS authority servers preserve the mixed case encoding of the query in answer messages. Attackers hoping to poison a DNS cache must therefore guess the mixed-case encoding of the query, in addition to all other fields required in a DNS poisoning attack. This increases the difficulty of the attack. We describe and measure the additional protections realized by this technique. Our analysis includes a basic model of DNS poisoning, measurement of the benefits that come from case-sensitive query encoding, implementation of the system for recursive DNS servers, and large-scale real-world experimental evaluation. Since the benefits of our technique can be significant, we have simultaneously made this DNS encoding system a proposed IETF standard. Our approach is practical enough that, just weeks after its disclosure, it is being implemented by numerous DNS vendors.

Towards complete node enumeration in a peer-to-peer botnet

Modern advanced botnets may employ a decentralized peer-to-peer overlay network to bootstrap and maintain their command and control channels, making them more resilient to traditional mitigation efforts such as server incapacitation. As an alternative strategy, the malware defense community has been trying to identify the bot-infected hosts and enumerate the IP addresses of the participating nodes so that the list can be used by system administrators to identify local infections, block spam emails sent from bots, and configure firewalls to protect local users. Enumerating the infected hosts, however, has presented challenges. One cannot identify infected hosts behind firewalls or NAT devices by employing crawlers, a commonly used enumeration technique where recursive get-peerlist lookup requests are sent newly discovered IP addresses of infected hosts. As many bot-infected machines in homes or offices are behind firewall or NAT devices, these crawler-based enumeration methods would miss a large portions of botnet infections. In this paper, we present the Passive P2P Monitor (PPM), which can enumerate the infected hosts regardless whether or not they are behind a firewall or NAT. As an empirical study, we examined the Storm botnet and enumerated its infected hosts using the PPM. We also improve our PPM design by incorporating a FireWall Checker (FWC) to identify nodes behind a firewall. Our experiment with the peer-to-peer Storm botnet shows that more than 40% of bots that contact the PPM are behind firewall or NAT devices, implying that crawler-based enumeration techniques would miss out a significant portion of the botnet population. Finally, we show that the PPMs coverage is based on a probability-based coverage model that we derived from the empirical observation of the Storm botnet.

Using labeling to prevent cross-service attacks against smart phones

Wireless devices that integrate the functionality of PDAs and cell phones are becoming commonplace, making different types of network services available to mobile applications. However, the integration of different services allows an attacker to cross service boundaries. For example, an attack carried out through the wireless network interface may eventually provide access to the phone functionality. This type of attacks can cause considerable damage because some of the services (e.g., the GSM-based services) charge the user based on the traffic or time of use. In this paper, we demonstrate the feasibility of these attacks by developing a proof-of-concept exploit that crosses service boundaries. To address these security issues, we developed a solution based on resource labeling. We modified the kernel of an integrated wireless device so that processes and files are marked in a way that allows one to regulate the access to different system resources. Labels are set when certain network services are accessed. The labeling is then transferred between processes and system resources as a result of either access or execution. We also defined a language for creating labeling rules, and demonstrated how the system can be used to prevent attacks that attempt to cross service boundaries. Experimental evaluation shows that the implementation introduces little overhead. Our security solution is orthogonal to other protection schemes and provides a critical defense for the growing problem of cell phone viruses and worms

Towards an information-theoretic framework for analyzing intrusion detection systems

IDS research still needs to strengthen mathematical foundations and theoretic guidelines. In this paper, we build a formal framework, based on information theory, for analyzing and quantifying the effectiveness of an IDS. We firstly present a formal IDS model, then analyze it following an information-theoretic approach. Thus, we propose a set of information-theoretic metrics that can quantitatively measure the effectiveness of an IDS in terms of feature representation capability, classification information loss, and overall intrusion detection capability. We establish a link to relate these metrics, and prove a fundamental upper bound on the intrusion detection capability of an IDS. Our framework is a practical theory which is data trace driven and evaluation oriented in this area. In addition to grounding IDS research on a mathematical theory for formal study, this framework provides practical guidelines for IDS fine-tuning, evaluation and design, that is, the provided set of metrics greatly facilitates a static/dynamic fine-tuning of an IDS to achieve optimal operation and a fine-grained means to evaluate IDS performance and improve IDS design. We conduct experiments to demonstrate the utility of our framework in practice.