Hing-Fung Ting

A simpler and more efficient deterministic scheme for finding frequent items over sliding windows

In this paper, we give a simple scheme for identifying ε-approximate frequent items over a sliding window of size <i>n</i>. Our scheme is deterministic and does not make any assumption on the distribution of the item frequencies. It supports <i>O</i>(1/ε) update and query time, and uses <i>O</i>(1/ε) space. It is very simple; its main data structures are just a few short queues whose entries store the position of some items in the sliding window. We also extend our scheme for variable-size window. This extended scheme uses <i>O</i>(1/ε log(ε<i>n</i>)) space.

SOAP3-dp: Fast, Accurate and Sensitive GPU-Based Short Read Aligner

To tackle the exponentially increasing throughput of Next-Generation Sequencing (NGS), most of the existing short-read aligners can be configured to favor speed in trade of accuracy and sensitivity. SOAP3-dp, through leveraging the computational power of both CPU and GPU with optimized algorithms, delivers high speed and sensitivity simultaneously. Compared with widely adopted aligners including BWA, Bowtie2, SeqAlto, CUSHAW2, GEM and GPU-based aligners BarraCUDA and CUSHAW, SOAP3-dp was found to be two to tens of times faster, while maintaining the highest sensitivity and lowest false discovery rate (FDR) on Illumina reads with different lengths. Transcending its predecessor SOAP3, which does not allow gapped alignment, SOAP3-dp by default tolerates alignment similarity as low as 60%. Real data evaluation using human genome demonstrates SOAP3-dps power to enable more authentic variants and longer Indels to be discovered. Fosmid sequencing shows a 9.1% FDR on newly discovered deletions. SOAP3-dp natively supports BAM file format and provides the same scoring scheme as BWA, which enables it to be integrated into existing analysis pipelines. SOAP3-dp has been deployed on Amazon-EC2, NIH-Biowulf and Tianhe-1A.

MEGAHIT v1.0: A fast and scalable metagenome assembler driven by advanced methodologies and community practices

The study of metagenomics has been much benefited from low-cost and high-throughput sequencing technologies, yet the tremendous amount of data generated make analysis like de novo assembly to consume too much computational resources. In late 2014 we released MEGAHIT v0.1 (together with a brief note of Li et al. (2015) [1]), which is the first NGS metagenome assembler that can assemble genome sequences from metagenomic datasets of hundreds of Giga base-pairs (bp) in a time- and memory-efficient manner on a single server. The core of MEGAHIT is an efficient parallel algorithm for constructing succinct de Bruijn Graphs (SdBG), implemented on a graphical processing unit (GPU). The software has been well received by the assembly community, and there is interest in how to adapt the algorithms to integrate popular assembly practices so as to improve the assembly quality, as well as how to speed up the software using better CPU-based algorithms (instead of GPU). In this paper we first describe the details of the core algorithms in MEGAHIT v0.1, and then we show the new modules to upgrade MEGAHIT to version v1.0, which gives better assembly quality, runs faster and uses less memory. For the Iowa Prairie Soil dataset (252Gbp after quality trimming), the assembly quality of MEGAHIT v1.0, when compared with v0.1, has a significant improvement, namely, 36% increase in assembly size and 23% in N50. More interestingly, MEGAHIT v1.0 is no slower than before (even running with the extra modules). This is primarily due to a new CPU-based algorithm for SdBG construction that is faster and requires less memory. Using CPU only, MEGAHIT v1.0 can assemble the Iowa Prairie Soil sample in about 43h, reducing the running time of v0.1 by at least 25% and memory usage by up to 50%. MEGAHIT v1.0, exhibiting a smaller memory footprint, can process even larger datasets. The Kansas Prairie Soil sample (484Gbp), the largest publicly available dataset, can now be assembled using no more than 500GB of memory in 7.5days. The assemblies of these datasets (and other large metgenomic datasets), as well as the software, are available at the website https://hku-bal.github.io/megabox.

New Results on On-Demand Broadcasting with Deadline via Job Scheduling with Cancellation

This paper studies the on-demand broadcasting problem with deadlines. We give the first general upper bound and improve existing lower bounds on the competitive ratio of the problem. The novelty of our work is the introduction of a new job scheduling problem that allows cancellation. We prove that the broadcasting problem can be reduced to this scheduling problem. This reduction frees us from the complication of the broadcasting model and allows us to work on a conceptually simpler model for upper bound results.

Continuous Monitoring of Distributed Data Streams over a Time-Based Sliding Window

In this paper we extend the study of algorithms for monitoring distributed data streams from whole data streams to a time-based sliding window. The concern is how to minimize the communication between individual streams and the root, while allowing the root, at any time, to report the global statistics of all streams within a given error bound. This paper presents communication-efficient algorithms for three classical statistics, namely, basic counting, frequent items and quantiles. The worst-case communication cost over a window is

Sleep with Guilt and Work Faster to Minimize Flow Plus Energy

O(\frac{k}{\varepsilon} \log\frac{\varepsilon N}{k})

Requirement-based data cube schema design

bits for basic counting,

Maintaining significant stream statistics over sliding windows

O(\frac{k}{\varepsilon} \log\frac{N}{k})

A new upper bound 2.5545 on 2D Online Bin Packing

words for frequent items and

General techniques for comparing unrooted evolutionary trees

O(\frac{k}{\varepsilon^{2}} \log\frac{N}{k})