Bertrand Anckaert

Software protection through dynamic code mutation

Reverse engineering of executable programs, by disassembling them and then using program analyses to recover high level semantic information, plays an important role in attacks against software systems, and can facilitate software piracy. This paper introduces a novel technique to complicate reverse engineering. The idea is to change the program code repeatedly as it executes, thereby thwarting correct disassembly. The technique can be made as secure as the least secure component of opaque variables and pseudorandom number generators.

Program obfuscation: a quantitative approach

Despite the recent advances in the theory underlying obfuscation, there still is a need to evaluate the quality of practical obfuscating transformations more quickly and easily. This paper presents the first steps toward a comprehensive evaluation suite consisting of a number of deobfuscating transformations and complexity metrics that can be readily applied on existing and future transformations in the domain of binary obfuscation. In particular, a framework based on software complexity metrics measuring four program properties: code, control flow, data and data flow is suggested. A number of well-known obfuscating and deobfuscating transformations are evaluated based upon their impact on a set of complexity metrics. This enables us to quantitatively evaluate the potency of the (de)obfuscating transformations.

Hybrid static-dynamic attacks against software protection mechanisms

Advances in reverse engineering and program analyses have made software extremely vulnerable to malicious host attacks. These attacks typically take the form of intellectual property violations, against which the software needs to be protected. The intellectual property that needs to be protected can take on different forms. The software might, e.g., consist itself of proprietary algorithms and datastructures or it could provide controlled access to copyrighted material. Therefore, in recent years, a number of techniques have been explored to protect software. Many of these techniques provide a reasonable level of security against static-only attacks. Many of them however fail to address the problem of dynamic or hybrid static-dynamic attacks. While this type of attack is already commonly used by black-hats, this is one of the first scientific papers to discuss the potential of these attacks through which an attacker can analyze, control and modify a program extensively. The concepts are illustrated through a case study of a recently proposed algorithm for software watermarking [6].

Software piracy prevention through diversity

Software piracy is a major concern for software providers, despite the many defense mechanisms that have been proposed to prevent it. This paper identifies the fundamental weaknesses of existing approaches, resulting from the static nature of defense and the impossibility to prevent the duplication of digital data. A new scheme is presented that enables a more dynamic nature of defense and makes it harder to create an additional, equally useful copy. Furthermore it enables a fine-grained control over the distributed software. Its strength is based on diversity: each installed copy is unique and updates are tailored to work for one installed copy only.

Towards tamper resistant code encryption: practice and experience

In recent years, many have suggested to apply encryption in the domain of software protection against malicious hosts. However, little information seems to be available on the implementation aspects or cost of the different schemes. This paper tries to fill the gap by presenting our experience with several encryption techniques: bulk encryption, an on-demand decryption scheme, and a combination of both techniques. Our scheme offers maximal protection against both static and dynamic code analysis and tampering. We validate our techniques by applying them on several benchmark programs of the CPU2006 Test Suite. And finally, we propose a heuristic which trades off security versus performance, resulting in a decrease of the runtime overhead.

Proteus: virtualization for diversified tamper-resistance

Despite huge efforts by software providers, software protection mechanisms are still broken on a regular basis. Due to the current distribution model, an attack against one copy of the software can be reused against any copy of the software. Diversity is an important tool to overcome this problem. It allows for renewable defenses in space, by giving every user a different copy, and renewable defenses in time when combined with tailored updates. This paper studies the possibilities and limitations of using virtualization to open a new set of opportunities to make diverse copies of a piece of software and to make individual copies more tamper-resistant. The performance impact is considerable and indicates that these techniques are best avoided in performance-critical parts of the code.

Steganography for executables and code transformation signatures

Steganography embeds a secret message in an innocuous cover-object. This paper identifies three cover-specific redundancies of executable programs and presents steganographic techniques to exploit these redundancies. A general framework to evaluate the stealth of the proposed techniques is introduced and applied on an implementation for the IA-32 architecture. This evaluation proves that, whereas existing tools such as Hydan [1] are insecure, significant encoding rates can in fact be achieved at a high security level.

Link-Time Optimization of IA64 Binaries

The features of the IA64 architecture create new opportunities for link-time optimization. At the same time they complicate the design of a link-time optimizer. This paper examines how to exploit some of the opportunities for link-time optimization and how to deal with the complications. The prototype link-time optimizer that implements the discussed techniques is able to reduce the code size of statically linked programs with 19% and achieves a speedup of 5.4% on average.

A model for self-modifying code

Self-modifying code is notoriously hard to understand and therefore very well suited to hide program internals. In this paper we introduce a program representation for this type of code: the state-enhanced control flow graph. It is shown how this program representation can be constructed, how it can be linearized into a binary program, and how it can be used to generate, analyze and transform self-modifying code.

Instruction Set Limitation in Support of Software Diversity

This paper proposes a novel technique, called instruction set limitation, to strengthen the resilience, of software. diversification against collusion attacks. Such attacks require a tool to match corresponding program fragments in different, diversified program versions. The proposed technique limits the types of instructions occurring in a program to the, most frequently occurring types; by replacing the infrequently used types as much as possible by more frequently used ones. As such, this technique, when combined with diversification techniques. reduces the number of easily matched code fragments. The proposed technique is evaluated against a powerful diversification tool for Intels x86 and an optimized matching process on a number of SPEC 2006 benchmarks.