Kunihiko Sadakane

MEGAHIT: an ultra-fast single-node solution for large and complex metagenomics assembly via succinct de Bruijn graph

MEGAHIT is a NGS de novo assembler for assembling large and complex metagenomics data in a time- and cost-efficient manner. It finished assembling a soil metagenomics dataset with 252 Gbps in 44.1 and 99.6 h on a single computing node with and without a graphics processing unit, respectively. MEGAHIT assembles the data as a whole, i.e. no pre-processing like partitioning and normalization was needed. When compared with previous methods on assembling the soil data, MEGAHIT generated a three-time larger assembly, with longer contig N50 and average contig length; furthermore, 55.8% of the reads were aligned to the assembly, giving a fourfold improvement.

Compressed Suffix Trees with Full Functionality

AbstractWe introduce new data structures for compressed suffix trees whose size are linear in the text size. The size is measured in bits; thus they occupy only O(n log|A|) bits for a text of length n on an alphabet A. This is a remarkable improvement on current suffix trees which require O(n log n) bits. Though some components of suffix trees have been compressed, there is no linear-size data structure for suffix trees with full functionality such as computing suffix links, string-depths and lowest common ancestors. The data structure proposed in this paper is the first one that has linear size and supports all operations efficiently. Any algorithm running on a suffix tree can also be executed on our compressed suffix trees with a slight slowdown of a factor of polylog(n).

New text indexing functionalities of the compressed suffix arrays

New text indexing functionalities of the compressed suffix arrays are proposed. The compressed suffix array proposed by Grossi and Vitter is a space-efficient data structure for text indexing. It occupies only O(n) bits for a text of length η; however it also uses the text itself that occupies n log2 |A| bits for the alphabet A. In this paper we modify the data structure so that pattern matching can be done without any access to the text. In addition to the original functions of the compressed suffix array, we add new operations search, decompress and inverse to the compressed suffix arrays. We show that the new index can find occ occurrences of any substring P of the text in O(|P| log n + occ loge n) time for any fixed 1 ≥ e > 0 without access to the text. The index also can decompress a part of the text of length m in O(m + loge n) time. For a text of length n on an alphabet A such that |A| = polylog(n), our new index occupies only O(nH0 + n log log |A|) bits where H0 ≤ log |A| is the order-0 entropy of the text. Especially for e = 1 the size is nH0 + O(n log log |A|) bits. Therefore the index will be smaller than the text, which means we can perform fast queries from compressed texts.

Succinct data structures for flexible text retrieval systems

We propose succinct data structures for text retrieval systems supporting document listing queries and ranking queries based on the tf*idf (term frequency times inverse document frequency) scores of documents. Traditional data structures for these problems support queries only for some predetermined keywords. Recently Muthukrishnan proposed a data structure for document listing queries for arbitrary patterns at the cost of data structure size. For computing the tf*idf scores there has been no efficient data structures for arbitrary patterns. Our new data structures support these queries using small space. The space is only 2/@e times the size of compressed documents plus 10n bits for a document collection of length n, for any 0<@e=<1. This is much smaller than the previous O(nlogn) bit data structures. Query time is O(m+qlog^@en) for listing and computing tf*idf scores for all q documents containing a given pattern of length m. Our data structures are flexible in a sense that they support queries for arbitrary patterns.

Compressed Text Databases with Efficient Query Algorithms Based on the Compressed Suffix Array

A compressed text database based on the compressed suffix array is proposed. The compressed suffix array of Grossi and Vitter occupies only O(n) bits for a text of length n; however it also uses the text itself that occupies O(n log |Σ|) bits for the alphabet Σ. On the other hand, our data structure does not use the text itself, and supports important operations for text databases: inverse, search and decompress. Our algorithms can find occ occurrences of any substring P of the text in O(|P| log n + occ logƐ n) time and decompress a part of the text of length l in O(l + logƐ n) time for any given 1 ≥ Ɛ > 0. Our data structure occupies only n(2/Ɛ (3/2 + H0 + 2 log H0) + 2 + 4 logƐ n/logƐ n-1)+o(n)+O(|Σ| log |Σ|) bits where H0 ≤ log |Σ| is the order-0 entropy of the text. We also show the relationship with the opportunistic data structure of Ferragina and Manzini.

Fully Functional Static and Dynamic Succinct Trees

We propose new succinct representations of ordinal trees and match various space/time lower bounds. It is known that any <i>n</i>-node static tree can be represented in 2<i>n</i> + <i>o</i>(<i>n</i>) bits so that a number of operations on the tree can be supported in constant time under the word-RAM model. However, the data structures are complicated and difficult to dynamize. We propose a simple and flexible data structure, called the <i>range min-max tree</i>, that reduces the large number of relevant tree operations considered in the literature to a few primitives that are carried out in constant time on polylog-sized trees. The result is extended to trees of arbitrary size, retaining constant time and reaching 2<i>n</i> + <i>O</i>(<i>n</i>/polylog(<i>n</i>)) bits of space. This space is optimal for a core subset of the operations supported and significantly lower than in any previous proposal. For the dynamic case, where insertion/deletion (indels) of nodes is allowed, the existing data structures support a very limited set of operations. Our data structure builds on the range min-max tree to achieve 2<i>n</i> + <i>O</i>(<i>n</i>/log <i>n</i>) bits of space and <i>O</i>(log <i>n</i>) time for all operations supported in the static scenario, plus indels. We also propose an improved data structure using 2<i>n</i> + <i>O</i>(<i>n</i>log log <i>n</i>/log <i>n</i>) bits and improving the time to the optimal <i>O</i>(log <i>n</i>/log log <i>n</i>) for most operations. We extend our support to forests, where whole subtrees can be attached to or detached from others, in time <i>O</i>(log^1+ε <i>n</i>) for any ε > 0. Such operations had not been considered before. Our techniques are of independent interest. An immediate derivation yields an improved solution to range minimum/maximum queries where consecutive elements differ by ± 1, achieving <i>n</i> + <i>O</i>(<i>n</i>/polylog(<i>n</i>)) bits of space. A second one stores an array of numbers supporting operations <i>sum</i> and <i>search</i> and limited updates, in optimal time <i>O</i>(log <i>n</i>/log log <i>n</i>). A third one allows representing dynamic bitmaps and sequences over alphabets of size σ, supporting rank/select and indels, within zero-order entropy bounds and time <i>O</i>(log <i>n</i> log σ/(log log <i>n</i>)²) for all operations. This time is the optimal <i>O</i>(log <i>n</i>/log log <i>n</i>) on bitmaps and polylog-sized alphabets. This improves upon the best existing bounds for entropy-bounded storage of dynamic sequences, compressed full-text self-indexes, and compressed-space construction of the Burrows-Wheeler transform.

Breaking a time-and-space barrier in constructing full-text indices

Suffix trees and suffix arrays are the most prominent full-text indices, and their construction algorithms are well studied. It has been open for a long time whether these indices can be constructed in both O(n log n) time and O(n log n)-bit working space, where n denotes the length of the text. In the literature, the fastest algorithm runs in O(n) time, while it requires O(n log n)-bit working space. On the other hand, the most space-efficient algorithm requires O(n)-bit working space while it runs in O(n log n) time. This paper breaks the long-standing time-and-space barrier under the unit-cost word RAM. We give an algorithm for constructing the suffix array which takes O(n) time and O(n)-bit working space, for texts with constant-size alphabets. Note that both the time and the space bounds are optimal. For constructing the suffix tree, our algorithm requires O(n log/sup /spl epsi//n) time and O(n)-bit working space for any 0 < /spl epsi/ < 1. Apart from that, our algorithm can also be adopted to build other existing full-text indices, such as Compressed Suffix Tree, Compressed Suffix Arrays and FM-index. We also study the general case where the size of the alphabet A is not constant. Our algorithm can construct a suffix array and a suffix tree using optimal O(n log |A|)-bit working space while running in O(n log log |A|) time and O(n log/sup /spl epsi//n) time, respectively. These are the first algorithms that achieve 0(n log n) time with optimal working space, under a reasonable assumption that log |A| = o(log n).

Squeezing succinct data structures into entropy bounds

Consider a sequence <i>S</i> of <i>n</i> symbols drawn from an alphabet <i>A</i> = {1, 2,. . .,σ}, stored as a binary string of <i>n</i>log σ bits. A <i>succinct</i> data structure on <i>S</i> supports a given set of primitive operations on <i>S</i> using just <i>f (n)</i> = <i>o</i>(<i>n</i> log σ) extra bits. We present a technique for transforming succinct data structures (which do not change the binary content of <i>S</i>) into <i>compressed</i> data structures using <i>nH</i><inf><i>k</i></inf> + <i>f(n)</i> + <i>O</i>(<i>n</i> log σ + log log<inf>σ</inf> <i>n</i> + <i>k</i>)/ log<inf>σ</inf> <i>n</i>) bits of space, where <i>H</i><inf><i>k</i></inf> ≤ log σ is the <i>k</i>th-order empirical entropy of <i>S</i>. When <i>k</i> + log σ = o(log <i>n</i>), we improve the space complexity of the succinct data structure from <i>n</i> log σ + <i>o</i>(<i>n</i> log σ) to <i>n</i> <i>H</i><inf><i>k</i></inf> + <i>o</i>(<i>n</i>log σ) bits by keeping <i>S</i> in compressed format, so that any substring of <i>O</i>(log σ <i>n</i>) symbols in <i>S</i> (i.e. <i>O</i>(log <i>n</i>) bits) can be decoded on the fly in constant time. Thus, the time complexity of the supported operations does not change asymptotically. Namely, if an operation takes <i>t</i>(<i>n</i>) time in the succinct data structure, it requires <i>O</i>(<i>t</i>(<i>n</i>)) time in the resulting compressed data structure. Using this simple approach we improve the space complexity of some of the best known results on succinct data structures We extend our results to handle another definition of entropy.

A Space and Time Efficient Algorithm for Constructing Compressed Suffix Arrays

With the first human DNA being decoded into a sequence of about 2.8 billion characters, much biological research has been centered on analyzing this sequence. Theoretically speaking, it is now feasible to accommodate an index for human DNA in the main memory so that any pattern can be located efficiently. This is due to the recent breakthrough on compressed suffix arrays, which reduces the space requirement from O(n log n) bits to O(n) bits. However, constructing compressed suffix arrays is still not an easy task because we still have to compute suffix arrays first and need a working memory of O(n log n) bits (i.e., more than 13 gigabytes for human DNA). This paper initiates the study of constructing compressed suffix arrays directly from the text. The main contribution is a construction algorithm that uses only O(n) bits of working memory, and the time complexity is O(n log n). Our construction algorithm is also time and space efficient for texts with large alphabets such as Chinese or Japanese. Precisely, when the alphabet size is |Σ|, the working space is O(n log |Σ|) bits, and the time complexity remains O(n log n), which is independent of |Σ|.

Compressed indexes for dynamic text collections

Let <i>T</i> be a string with <i>n</i> characters over an alphabet of constant size. A recent breakthrough on compressed indexing allows us to build an index for <i>T</i> in optimal space (i.e., <i>O</i>(<i>n</i>) bits), while supporting very efficient pattern matching [Ferragina and Manzini 2000; Grossi and Vitter 2000]. Yet the compressed nature of such indexes also makes them difficult to update dynamically. This article extends the work on optimal-space indexing to a dynamic collection of texts. Our first result is a compressed solution to the <i>library management</i> problem, where we show an index of <i>O</i>(<i>n</i>) bits for a text collection <i>L</i> of total length <i>n</i>, which can be updated in <i>O</i>(|<i>T</i>| log <i>n</i>) time when a text <i>T</i> is inserted or deleted from <i>L</i>; also, the index supports searching the occurrences of any pattern <i>P</i> in all texts in <i>L</i> in <i>O</i>(|<i>P</i>| log <i>n</i> + <i>occ</i> log² <i>n</i>) time, where <i>occ</i> is the number of occurrences. Our second result is a compressed solution to the <i>dictionary matching</i> problem, where we show an index of <i>O</i>(<i>d</i>) bits for a pattern collection <i>D</i> of total length <i>d</i>, which can be updated in <i>O</i>(|<i>P</i>| log² <i>d</i>) time when a pattern <i>P</i> is inserted or deleted from <i>D</i>; also, the index supports searching the occurrences of all patterns of <i>D</i> in any text <i>T</i> in <i>O</i>((|<i>T</i>| + <i>occ</i>)log² <i>d</i>) time. When compared with the <i>O</i>(<i>d</i> log <i>d</i>)-bit suffix-tree-based solution of Amir et al. [1995], the compact solution increases the query time by roughly a factor of log <i>d</i> only. The solution to the dictionary matching problem is based on a new compressed representation of a suffix tree. Precisely, we give an <i>O</i>(<i>n</i>)-bit representation of a suffix tree for a dynamic collection of texts whose total length is <i>n</i>, which supports insertion and deletion of a text <i>T</i> in <i>O</i>(|<i>T</i>| log² <i>n</i>) time, as well as all suffix tree traversal operations, including forward and backward suffix links. This work can be regarded as a generalization of the compressed representation of static texts. In the study of the aforementioned result, we also derive the first <i>O</i>(<i>n</i>)-bit representation for maintaining <i>n</i> pairs of balanced parentheses in <i>O</i>(log <i>n</i>/log log <i>n</i>) time per operation, matching the time complexity of the previous <i>O</i>(<i>n</i> log <i>n</i>)-bit solution.