Lok Kwong Yan

V2E: combining hardware virtualization and softwareemulation for transparent and extensible malware analysis

A transparent and extensible malware analysis platform is essential for defeating malware. This platform should be transparent so malware cannot easily detect and bypass it. It should also be extensible to provide strong support for heavyweight instrumentation and analysis efficiency. However, no existing platform can meet both requirements. Leveraging hardware virtualization technology, analysis platforms like Ether can achieve good transparency, but its instrumentation support and analysis efficiency is poor. In contrast, software emulation provides strong support for code instrumentation and good analysis efficiency by using dynamic binary translation. However, analysis platforms based on software emulation can be easily detected by malware and thus is poor in transparency. To achieve both transparency and extensibility, we propose a new analysis platform that combines hardware virtualization and software emulation. The essence is precise heterogeneous replay: the malware execution is recorded via hardware virtualization and then replayed in software. Our design ensures the execution replay is precise. Moreover, with page-level recording granularity, the platform can easily adjust to analyze various forms of malware (a process, a kernel module, or a shared library). We implemented a prototype called V2E and demonstrated its capability and efficiency by conducting an extensive evaluation with both synthetic samples and 14 realworld emulation-resistant malware samples.

Make it work, make it right, make it fast: building a platform-neutral whole-system dynamic binary analysis platform

Dynamic binary analysis is a prevalent and indispensable technique in program analysis. While several dynamic binary analysis tools and frameworks have been proposed, all suffer from one or more of: prohibitive performance degradation, semantic gap between the analysis code and the program being analyzed, architecture/OS specificity, being user-mode only, lacking APIs, etc. We present DECAF, a virtual machine based, multi-target, whole-system dynamic binary analysis framework built on top of QEMU. DECAF provides Just-In-Time Virtual Machine Introspection combined with a novel TCG instruction-level tainting at bit granularity, backed by a plugin based, simple-to-use event driven programming interface. DECAF exercises fine control over the TCG instructions to accomplish on-the-fly optimizations. We present 3 platform-neutral plugins - Instruction Tracer, Keylogger Detector, and API Tracer, to demonstrate the ease of use and effectiveness of DECAF in writing cross-platform and system-wide analysis tools. Implementation of DECAF consists of 9550 lines of C++ code and 10270 lines of C code and we evaluate DECAF using CPU2006 SPEC benchmarks and show average overhead of 605% for system wide tainting and 12% for VMI.

Virtual honeynets revisited

A new user-mode Linux based virtual honeynet architecture is presented in this paper. The new architecture has improved functionality that is difficult to realize in the GenII honeynet. Two new honeynet capabilities in particular are introduced. Honeypot controller is a new virtual honeynet component that assists in data control. The honeywall promises to have finer control over the honeypots through signal and system call redirections. The second new capability is the disk imager. The disk imager is capable of making forensic images of the virtual machines file systems for further analysis. Since security for virtual honeynets is a big concern, the new virtual honeynet architecture utilizes security enhanced Linux to isolate the untrusted honeypots from the completely trusted honeywall. SELinux and other research work done in this field made the new honeynet architecture a viable alternative to GenII honeynets. A file system logging mechanism, FSLog, has been developed for the UML based virtual honeynet. In conjunction with the built-in tty logger, UML based honeynets have logging capabilities that are equivalent to their GenII honeynet counterparts. The current version of FSLog successfully logs eighteen virtual file systems system calls including the common, read(), write(), open() and close() functions. Its current functionality and how it pieces into the new architecture is also discussed. This work provides researchers with an alternative honeynet platform. The new virtual honeynet architecture is more portable, easier to setup, more cost effective and as secure as the GenII honeynet. The addition of the honeypot controller and disk imager components also makes the new virtual honeynet architecture more capable.

DECAF: A Platform-Neutral Whole-System Dynamic Binary Analysis Platform

Dynamic binary analysis is a prevalent and indispensable technique in program analysis. While several dynamic binary analysis tools and frameworks have been proposed, all suffer from one or more of: prohibitive performance degradation, a semantic gap between the analysis code and the program being analyzed, architecture/OS specificity, being user-mode only, and lacking APIs. We present DECAF, a virtual machine based, multi-target, whole-system dynamic binary analysis framework built on top of QEMU. DECAF provides Just-In-Time Virtual Machine Introspection and a plugin architecture with a simple-to-use event-driven programming interface. DECAF implements a new instruction-level taint tracking engine at bit granularity, which exercises fine control over the QEMU Tiny Code Generator (TCG) intermediate representation to accomplish on-the-fly optimizations while ensuring that the taint propagation is sound and highly precise. We perform a formal analysis of DECAFs taint propagation rules to verify that most instructions introduce neither false positives nor false negatives. We also present three platform-neutral plugins—Instruction Tracer, Keylogger Detector, and API Tracer, to demonstrate the ease of use and effectiveness of DECAF in writing cross-platform and system-wide analysis tools. Implementation of DECAF consists of 9,550 lines of C++ code and 10,270 lines of C code and we evaluate DECAF using CPU2006 SPEC benchmarks and show average overhead of 605 percent for system wide tainting and 12 percent for VMI.

Dynamic VM Dependability Monitoring Using Hypervisor Probes

Many current VM monitoring approaches require guest OS modifications and are also unable to perform application level monitoring, reducing their value in a cloud setting. This paper introduces hprobes, a framework that allows one to dynamically monitor applications and operating systems inside a VM. The hprobe framework does not require any changes to the guest OS, which avoids the tight coupling of monitoring with its target. Furthermore, the monitors can be customized and enabled/disabled while the VM is running. To demonstrate the usefulness of this framework, we present three sample detectors: an emergency detector for a security vulnerability, an application watchdog, and an infinite-loop detector. We test our detectors on real applications and demonstrate that those detectors achieve an acceptable level of performance overhead with a high degree of flexibility.

MOSE: Live Migration Based On-the-Fly Software Emulation

Software emulation has been proven useful in many scenarios, such as software testing, malware analysis, and intrusion response. However, fine-grained software emulation (e.g., at the instruction level) incurs considerable execution overhead (about 8x performance degradation), which hampers its use in production settings. In this paper, we propose MOSE (Live Migration based On-the-fly Software Emulation) that combines the performance advantages of hardware virtualization and the fine-grained analysis capability (comprehensiveness) of whole-system software emulation. Namely, a system can run as normal on a hardware-virtualized platform at near native speed, but when needed, it can be live-migrated to an emulator, not necessarily running on the same physical system, for in-depth analysis and triage; when the analysis is complete, the virtual machine can be migrated back to benefit from full hardware-virtualization again. In this way, the performance degradation is only experienced during analysis and triage. To demonstrate this new capability, we built a proof of concept on-the-fly software emulation system, based on QEMU/KVM and DECAF, the Dynamic Executable Code Analysis Framework. We also perform three case studies: automated kernel panic triage, live-patching a security vulnerability, and on-demand symbolic execution, to illustrate on-demand instruction level analysis.

Using OS Design Patterns to Provide Reliability and Security as-a-Service for VM-based Clouds

This paper extends the concepts behind cloud services to offer hypervisor-based reliability and security monitors for cloud virtual machines. Cloud VMs can be heterogeneous and as such guest OS parameters needed for monitoring can vary across different VMs and must be obtained in some way. Past work involves running code inside the VM, which is unacceptable for a cloud environment. We solve this problem by recognizing that there are common OS design patterns that can be used to infer monitoring parameters from the guest OS. We extract information about the cloud users guest OS with the users existing VM image and knowledge of OS design patterns as the only inputs to analysis. To demonstrate the range of monitoring functionality possible with this technique, we implemented four sample monitors: a guest OS process tracer, an OS hang detector, a return-to-user attack detector, and a process-based keylogger detector.

Formal Modeling and Verification of Security Property in Handel C Program

Multi-million gate system-on-chip SoC designs easily fit into todays Field Programmable Gate Arrays FPGAs. As FPGAs become more common in safety-critical and mission-critical systems, researchers and designers require information flow guarantees for the FPGAs. Tools for designing a secure system of chips SOCs using FPGAs and new techniques to manage and analyze the security properties precisely are desirable. In this work we propose a formal approach to model, analyze and verify a typical set of security properties-noninterference-of Handel C programs using Petri Nets and model checking. This paper presents a method to model Handel C programs using Predicate Transition Nets, a type of Petri Net, and define security properties on the model, plus a verification approach where security properties are checked. Three steps are used. First, a formal specification on the Handel C description using Petri Nets is extracted. Second, the dynamic noninterference properties with respect to the Handel C program statements are defined on the model. To assist in verification, a translation rule from the Petri Nets specification to the Maude programming language is also defined. Thus, the formal specification can be verified against the system properties using model checking. A case study of the pipeline multiplier is discussed to illustrate the concept and validate the approach.