Charlotte Herzeel

Halvade: scalable sequence analysis with MapReduce.

Motivation: Post-sequencing DNA analysis typically consists of read mapping followed by variant calling. Especially for whole genome sequencing, this computational step is very time-consuming, even when using multithreading on a multi-core machine. Results: We present Halvade, a framework that enables sequencing pipelines to be executed in parallel on a multi-node and/or multi-core compute infrastructure in a highly efficient manner. As an example, a DNA sequencing analysis pipeline for variant calling has been implemented according to the GATK Best Practices recommendations, supporting both whole genome and whole exome sequencing. Using a 15-node computer cluster with 360 CPU cores in total, Halvade processes the NA12878 dataset (human, 100 bp paired-end reads, 50× coverage) in <3 h with very high parallel efficiency. Even on a single, multi-core machine, Halvade attains a significant speedup compared with running the individual tools with multithreading. Availability and implementation: Halvade is written in Java and uses the Hadoop MapReduce 2.0 API. It supports a wide range of distributions of Hadoop, including Cloudera and Amazon EMR. Its source is available at http://bioinformatics.intec.ugent.be/halvade under GPL license. Contact: jan.fostier@intec.ugent.be Supplementary information: Supplementary data are available at Bioinformatics online.

Fact spaces: coordination in the face of disconnection

Coordination languages for ad hoc networks with a fluid topology do not offer adequate support to detect and deal with device disconnection. Such a disconnection is particularly relevant if the device provided context information rather than emitting messages, as such context information then becomes invalid. This paper proposes the Fact Space Model which establishes a logic coordination language on top of LIMEs federated tuple space. In the model, the federated space offers applications a consistent view of their environment over which they can reason using logic rules. These rules encode which conclusions may be drawn from the presence of particular facts, and are similarly used to ensure the consistency of these conclusions when devices go out of range. By allowing applications to add application-specific hooks to these rules, the application programmer is offered a general-purpose mechanism to respond to the discovery and disconnection of devices.

Reflection for the Masses

A reflective programming language provides means to render explicit what is typically abstracted away in its language constructs in an on-demand style. In the early 1980s, Brian Smith introduced a general recipe for building reflective programming languages with the notion of procedural reflection . It is an excellent framework for understanding and comparing various metaprogramming and reflective approaches, including macro programming, first-class environments, first-class continuations, metaobject protocols, aspect-oriented programming, and so on. Unfortunately, the existing literature of Brian Smiths original account of procedural reflection is hard to understand: It is based on terminology derived from philosophy rather than computer science, and takes concepts for granted that are hard to reconstruct without intimate knowledge of historical Lisp dialects from the 1960s and 1970s. We attempt to untangle Smiths original account of procedural reflection and make it accessible to a new and wider audience. On the other hand, we then use its terminological framework to analyze other metaprogramming and reflective approaches, especially those that came afterwards.

Dynamic parallelization of recursive code: part 1: managing control flow interactions with the continuator

While most approaches to automatic parallelization focus on compilation approaches for parallelizing loop iterations, we advocate the need for new virtual machines that can parallelize the execution of recursive programs. In this paper, we show that recursive programs can be effectively parallelized when arguments to procedures are evaluated concurrently and branches of conditional statements are speculatively executed in parallel. We introduce the continuator concept, a runtime structure that tracks and manages the control dependences between such concurrently spawned tasks, ensuring adherence to the sequential semantics of the parallelized program. As a proof of concept, we discuss the details of a parallel interpreter for Scheme (implemented in Common Lisp) based on these ideas, and show the results from executing the Clinger benchmark suite for Scheme.

Filtered dispatch

Predicate dispatching is a generalized form of dynamic dispatch, which has strong limitations when arbitrary predicates of the underlying base language are used. Unlike classes, which enforce subset relationships between their sets of instances, arbitrary predicates generally do not designate subsets of each other, so methods whose applicability is based on predicates cannot be ordered according to their specificity in the general case. This paper introduces a decidable but expressive alternative mechanism called filtered dispatch that adds a simple preprocessing step before the actual method dispatch is performed and thus enables the use of arbitrary predicates for selecting and applying methods.

Resolving Load Balancing Issues in BWA on NUMA Multicore Architectures

Running BWA in multithreaded mode on a multi-socket server results in poor scaling behaviour. This is because the current parallelisation strategy does not take into account the load imbalance that is inherent to the properties of the data being aligned, e.g. varying read lengths and numbers of mutations. Additional load imbalance is also caused by the BWA code not anticipating certain hardware characteristics of multi-socket multicores, such as the non-uniform memory access time of the different cores. We show that rewriting the parallel section using Cilk removes the load imbalance, resulting in a factor two performance improvement over the original BWA.

Context-oriented software transactional memory in common lisp

Software transactional memory (STM) is a promising approach for coordinating concurrent threads, for which many implementation strategies are currently being researched. Although some first steps exist to ease experimenting with different strategies, this still remains a relatively complex and cumbersome task. The reason is that software transactions require STM-specific dynamic crosscutting adaptations, but this is not accounted for in current STM implementations. This paper presents CSTM, an STM framework based on Context-oriented Progamming, in which transactions are modelled as dynamically scoped layer activations. It enables expressing transactional variable accesses as user-defined crosscutting concerns, without requiring invasive changes in the rest of a program. This paper presents a proof-of-concept implementation based on ContextL for Common Lisp, along with example STM strategies and preliminary benchmarks, and introduces some of ContextLs unique features for context-dependent variable accesses.

Forward chaining in HALO: An implementation strategy for history-based logic pointcuts

In aspect-oriented programming, pointcuts are formulated as conditions over the context of dynamic events in the execution of a program. Hybrid pointcut languages also allow this context to come from interactions between the pointcut language and the base program. While some pointcut languages only allow conditions on the current execution event, more recent proposals have demonstrated the need for expressing conditions over a history of join points. Such pointcut languages require means to balance the expressiveness of the language with the additional memory and runtime overhead caused by keeping a history of join point context data. In this paper, we introduce a logic-based pointcut language that allows interaction with the base program as well as pointcuts over a history of join points. We introduce forward chaining as an implementation model for this language, and discuss possible optimization strategies for the additional overhead.

elPrep: High-Performance Preparation of Sequence Alignment/Map Files for Variant Calling.

elPrep is a high-performance tool for preparing sequence alignment/map files for variant calling in sequencing pipelines. It can be used as a replacement for SAMtools and Picard for preparation steps such as filtering, sorting, marking duplicates, reordering contigs, and so on, while producing identical results. What sets elPrep apart is its software architecture that allows executing preparation pipelines by making only a single pass through the data, no matter how many preparation steps are used in the pipeline. elPrep is designed as a multithreaded application that runs entirely in memory, avoids repeated file I/O, and merges the computation of several preparation steps to significantly speed up the execution time. For example, for a preparation pipeline of five steps on a whole-exome BAM file (NA12878), we reduce the execution time from about 1:40 hours, when using a combination of SAMtools and Picard, to about 15 minutes when using elPrep, while utilising the same server resources, here 48 threads and 23GB of RAM. For the same pipeline on whole-genome data (NA12878), elPrep reduces the runtime from 24 hours to less than 5 hours. As a typical clinical study may contain sequencing data for hundreds of patients, elPrep can remove several hundreds of hours of computing time, and thus substantially reduce analysis time and cost.

Escaping with future variables in HALO

HALO is a novel aspect language introducing a logic-based pointcut language which combines history-based pointcuts and “escape” conditions for interacting with the base language. This combination is difficult to support when escape conditions access context exposed by “future” join points. This paper introduces a weaving mechanism based on copying objects for resolving such pointcuts. Though this seems a memory consuming solution, it can be easily combined with HALO’s analysis for reducing the join point history. Furthermore, pointcuts with escape conditions accessing future join point context, sometimes require less memory than pointcuts that don’t, but otherwise implement the same functionality. In this paper, we illustrate this by measuring memory usage for simulations of an e-commerce application, switching between an implementation where the pointcut definitions contain escape conditions referring to future join point context, and an equivalent implementation that doesn’t.