Steve Uurtamo

Data stream algorithms for codeword testing

Motivated by applications in storage systems and property testing, we study data stream algorithms for local testing and tolerant testing of codes. Ideally, we would like to know whether there exist asymptotically good codes that can be local/tolerant tested with one-pass, poly-log space data stream algorithms. We show that for the error detection problem (and hence, the local testing problem), there exists a one-pass, log-space data stream algorithm for a broad class of asymptotically good codes, including the Reed-Solomon (RS) code and expander codes. In our technically more involved result, we give a one-pass, O(e log2n)-space algorithm for RS (and related) codes with dimension k and block length n that can distinguish between the cases when the Hamming distance between the received word and the code is at most e and at least a ċ e for some absolute constant a > 1. For RS codes with random errors, we can obtain e ≤ O(n/k). For folded RS codes, we obtain similar results for worst-case errors as long as e ≤ (n/k)1-e for any constant e > 0. These results follow by reducing the tolerant testing problem to the error detection problem using results from group testing and the list decodability of the code. We also show that using our techniques, the space requirement and the upper bound of e ≤ O(n/k) cannot be improved by more than logarithmic factors.

Bidirectional data verification for cloud storage

This paper presents a storage enforcing remote verification scheme, PGV (Pretty Good Verification) as a bidirectional data integrity checking mechanism for cloud storage. At its core, PGV relies on the well-known polynomial hash; we show that the polynomial hash provably possesses the storage enforcement property and is also efficient in terms of performance. In addition to the traditional application of a client verifying the storage content at a remote server, PGV can also be applied to de-duplication scenarios where the server wants to verify whether the client possesses a significant amount of information about a file (and not just a partial knowledge/fingerprint of the file) before granting access to an existing file.While existing schemes are often developed to handle a malicious adversarial model, we argue that such a model is often too strong of an assumption, resulting in over-engineered, resource-intensive mechanisms. Instead, the storage enforcement property of PGV aims at removing a practical incentive for a storage server to cheat in order to save on storage space in a covert adversarial model.We theoretically prove the power of PGV by combining Kolmogorov complexity and list decoding and experimentally show the simplicity and low overhead of PGV by comparing it with existing schemes. Altogether, PGV provides a good, practical way to perform storage enforcing remote verification.

Two theorems on list decoding

We prove the following results concerning the list decoding of error-correcting codes: 1. We show that for any code with a relative distance of δ (over a large enough alphabet), the following result holds for random errors: With high probability, for a ρ ≤ δ - e fraction of random errors (for any e > 0), the received word will have only the transmitted codeword in a Hamming ball of radius ρ around it. Thus, for random errors, one can correct twice the number of errors uniquely correctable from worst-case errors for any code. A variant of our result also gives a simple algorithm to decode Reed-Solomon codes from random errors that, to the best of our knowledge, runs faster than known algorithms for certain ranges of parameters. 2. We show that concatenated codes can achieve the list decoding capacity for erasures. A similar result for worst-case errors was proven by Guruswami and Rudra (SODA 08), although their result does not directly imply our result. Our results show that a subset of the random ensemble of codes considered by Guruswami and Rudra also achieve the list decoding capacity for erasures. We also show that the exponential list size bound in our result with outer random linear codes cannot be improved using the recent techniques of Guruswami, Hastad and Kopparty that achieved similar improvements for errors. Our proofs employ simple counting and probabilistic arguments.

Polynomial fitting of data streams with applications to codeword testing

Given a stream of

PGV: A Storage Enforcing Remote Verification Scheme

(x,y)

Two Theorems in List Decoding

points, we consider the problem of finding univariate polynomials that best fit the data. Over finite fields, this problem encompasses the well-studied problem of decoding Reed-Solomon codes while over the reals it corresponds to the well-studied polynomial regression problem. We present one-pass algorithms for two natural problems: i) find the polynomial of a given degree

Almost Universal Hash Families are also Storage Enforcing

k

k + decision trees

that minimizes the error and ii) find the polynomial of smallest degree that interpolates through the points with at most a given error bound. We consider a range of error models including the average error per point, the maximum error, and the number of points that are not fitted exactly. Many of our results apply to both the reals and finite fields. As a consequence we also solve an open question regarding the tolerant testing of codes in the data stream model.

Two Theorems on List Decoding - (Extended Abstract).

This paper presents a storage enforcing remote verification scheme, PGV (Pretty Good Verification). While existing schemes are often developed to handle a malicious adversarial model, we argue that such a model is often too strong of an assumption, resulting in over-engineered, resource-intensive mechanisms. Instead, the storage enforcement property of PGV aims at removing a practical incentive for a storage server to cheat in order to save on storage space in a covert adversarial model. At its core, PGV relies on the well-known polynomial hash, we show that the polynomial hash provably possesses the storage enforcement property and is also efficient in terms of performance. In addition to the traditional application of a client verifying the storage content at a remote server, PGV can also be applied to de-duplication scenarios where the server wants to verify whether the client possesses a significant amount of information about a file (and not just a partial knowledge/fingerprint of the file) before granting access to an existing file. We theoretically prove the power of PGV by combining Kolmogorov complexity and list decoding, and experimentally show the simplicity and low overhead of PGV by comparing it with existing schemes. Altogether, PGV provides a good, practical way to perform storage enforcing remote verification.