icLibFuzzer: Isolated-context libFuzzer for Improving Fuzzer Comparability

Abstract

AFL [20] and libFuzzer [14] are two of the most successful fuzzers with different design goals and have been evolving separately for several years. AFL aims to continuously fuzz executables, while libFuzzer targets library functions and stops whenever hitting a crash. As a result, fuzzers in the AFL and libFuzzer families mainly consider the code-coverage and time-to-first-crash metrics, respectively. It is challenging to compare them directly with each other over the same metrics. This comparability issue is solved partially by enabling the ignore-crash mode in libFuzzer. The ignore-crash mode spawns multiple fuzzer instances and fuzzes a target executable in different processes parallelly. Whenever a fuzzer instance exits or finds a crash, libFuzzer will restart the fuzzer instance. Although libFuzzer in the ignore-crash mode can fuzz continuously and restart after a crash, the fuzzing results may be incorrect when previous non-crash runs pollute the global context because of libFuzzer’s in-process infrastructure. Therefore, most fuzzers were compared with libFuzzer only on few manually tuned datasets (e.g., fuzzer-test-suite [9]) that allow both fuzzer families to operate correctly. Due to the lack of comparability, the advances in libFuzzer seem to be left unnoticed in the state-of-the-art academic papers, which predominantly consider the code-coverage metric. For example, libFuzzer has integrated advanced instrumentation features such as CMP tracing since 2017, but similar features were unavailable in AFL-based fuzzers until 2019 in RedQueen [2] (which is later integrated into AFL++ [7]). To this end, we believe a bridge between these two fuzzer families is needed to foster knowledge accumulation and sharing in the community. In this work, we present isolated-context libFuzzer (icLibFuzzer), a new libFuzzer mode that bridges the two fuzzer families and allows them to be compared over commonly seen metrics on arbitrary programs. To solve the context pollution problem, icLibFuzzer runs in the AFL-like forkserver architecture and correctly manages the global context between executions. As speed is a crucial requirement of fuzzers, we further optimize icLibFuzzer’s implementation via structure packing, achieving similar performance to other forkserverbased implementation while preserving libFuzzer’s advanced instrumentation features. To demonstrate that icLibFuzzer improves fuzzer comparability, we compare icLibFuzzer with four state-of-the-art fuzzers (AFL, Angora, Honggfuzz, and QSYM) using the Abstract—libFuzzer is a powerful fuzzer that has helped find thousands of bugs in real-world programs. However, fuzzers that seek to compare with libFuzzer and its variants face two significant limitations. First, they are restricted to use the time-tofirst-crash metric rather than the code-coverage metric because libFuzzer will abort whenever the fuzzing target crashes. Second, even if libFuzzer in the ignore-crash mode can continue after finding a crash, it may produce wrong results for programs expecting a clean global context. Thus, fuzzers wishing to compare with libFuzzer are restricted to use carefully modified programs or programs without global-context dependency. To solve this context pollution problem and enhance comparability between libFuzzer and other fuzzers, we present a new libFuzzer mode called isolated-context mode (icLibFuzzer) that isolates the contexts of each fuzzer instance and fuzzing target, allowing to reinitialize the fuzzing target’s context after each execution efficiently. To implement icLibFuzzer, we modify libFuzzer’s inprocess infrastructure into a lightweight forkserver infrastructure inspired by AFL’s design and propose structure packing, which speeds up the fuzzing speed by about 2x. We compare icLibFuzzer with four state-of-the-art fuzzers (AFL, Angora, QSYM, and Honggfuzz) using several real-world programs. The experiment result shows that icLibFuzzer outperforms these four fuzzers in most target programs after 24 hours of fuzzing and maintains the lead from 24 to 72 hours. To demonstrate that we can easily keep up with libFuzzer’s updates, we upgrade icLibFuzzer to using the latest libFuzzer (from LLVM9 to LLVM11) with no change to our code base. Our preliminary evaluation hints at icLibFuzzer-LLVM11’s promising improvement compared with icLibFuzzer-LLVM9 and AFL++, one of the latest fuzzers in the AFL family. We hope icLibFuzzer can serve as another baseline for fuzzing research. Our source code is available at GitHub.