Compressed Sensing with 1D Total Variation: Breaking Sample Complexity Barriers via Non-Uniform Recovery (iTWIST'20)
CCompressed Sensing with 1D Total Variation: Breaking SampleComplexity Barriers via Non-Uniform Recovery
Martin Genzel , Maximilian M¨arz , and Robert Seidel Technische Universit¨at Berlin, Department of Mathematics, Berlin, Germany. Technische Universit¨at Berlin, Institute of Software Engineering and Theoretical Computer Science, Berlin, Germany.
Abstract—
This paper investigates total variation minimizationin one spatial dimension for the recovery of gradient-sparse sig-nals from undersampled Gaussian measurements. Recently es-tablished bounds for the required sampling rate state that uni-form recovery of all s -gradient-sparse signals in R n is only pos-sible with m (cid:38) √ sn · PolyLog( n ) measurements. Such acondition is especially prohibitive for high-dimensional problems,where s is much smaller than n . However, previous empiricalfindings seem to indicate that the latter sampling rate does notreflect the typical behavior of total variation minimization. In-deed, this work provides a rigorous analysis that breaks the √ sn -bottleneck for a large class of natural signals. The main resultshows that non-uniform recovery succeeds with high probabilityfor m (cid:38) s · PolyLog( n ) measurements if the jump discontinu-ities of the signal vector are sufficiently well separated. In partic-ular, this guarantee allows for signals arising from a discretizationof piecewise constant functions defined on an interval. The presentpaper serves as a short summary of the main results in our recentwork [GMS20]. We consider the following inverse problem: Assume that x ∗ ∈ R n denotes a signal of interest in one spatial dimension. It isassumed to be s -gradient-sparse , i.e., | supp( ∇ x ∗ ) | ≤ s , where ∇ ∈ R n − × n denotes a discrete gradient operator . Instead ofhaving direct access to x ∗ , the signal is observed via a linear,non-adaptive measurement process y = Ax ∗ ∈ R m , where A ∈ R m × n is a known measurement matrix . Themethodology of compressed sensing suggests that, under cer-tain conditions, it remains possible to retrieve x ∗ from theknowledge of y and A even when m (cid:28) n . Indeed, one of theseminal works of this field by Cand`es et al. [CRT06] shows thatfor random Fourier measurements, the recovery of x ∗ remainsfeasible with high probability as long as the number of mea-surements obeys m (cid:38) s log( n ) , where the ‘ (cid:38) ’-notation hides auniversal constant. For the success of this strategy, it is crucialto employ non-linear recovery methods that exploit the a prioriknowledge that x ∗ is gradient-sparse. Arguably, the most pop-ular version of 1D total variation (TV) minimization is basedon an adaption of the classical basis pursuit, i.e., one solves theconvex problem min x ∈ R n (cid:107)∇ x (cid:107) subject to y = Ax . (TV-1) We consider a gradient operator that is based on forward differences andvon Neumann boundary conditions. An extension to other choices is expectedto be straightforward. For the sake of simplicity, potential distortions in the measurement processare ignored here, but we emphasize that all results of this work can be maderobust against (adversarial) noise.
The research of the past three decades demonstrates that en-couraging a small TV norm often efficaciously reflects the in-herent structure of real-world signals. Although not as popu-lar as its counterpart in 2D (e.g., see [Cha04; CL97; ROF92]),TV methods in one spatial dimension find application in manypractical scenarios, e.g., see [LJ10; LJ11; PF16; SKBBH15;WWL14]. Furthermore, TV in 1D has frequently been subjectof mathematical research [BCNO11; Con13; Gra07; MG97;SPB15; Sel12].The main objective of this work is to study the 1D TV mini-mization problem for the benchmark case of Gaussian randommeasurements. In a nutshell, we intend to answer the followingquestion:
Assuming that A ∈ R m × n is a standard Gaussianrandom matrix, under which conditions is it possibleto recover an s -gradient-sparse signal x ∗ ∈ R n viaTV minimization (TV-1) with the near-optimal rate of m (cid:38) s · PolyLog( n ) measurements? At first sight, the aforementioned recovery result of Cand`es etal. [CRT06] seems to deny the relevance of the previous re-search question. However, we emphasize that their result ap-plies exclusively to random Fourier measurements. Indeed, theTV-Fourier combination allows for a significant simplificationof the problem, since the gradient operator is “compatible” withthe Fourier transform (differentiation is a Fourier multiplier).In contrast, the more recent work of Cai and Xu [CX15] ad-dresses the generic case of Gaussian measurements. However,their main result [CX15, Thm. 2.1] seems to imply a nega-tive answer to the question above: in essence, it shows thatthe uniform recovery of every s -gradient-sparse signal by solv-ing (TV-1) is possible if and only if the number of measure-ments obeys m (cid:38) √ sn · log( n ) . The conclusion from this result is as surprising as it is dis-couraging: It suggests that the threshold for successful re-covery of s -gradient-sparse signals via (TV-1) is essentiallygiven by √ sn -many Gaussian measurements. Remarkably, thelatter rate does not resemble the desirable standard criterion m (cid:38) s · PolyLog( n, s ) .In Table 1, we have summarized some of the existing guar-antees for TV minimization in compressed sensing. We referthe interested reader to [GMS20, Sec. 1.2] and [KKS17] for amore detailed overview of the relevant literature. a r X i v : . [ c s . I T ] S e p d D 1D ≥ Gaussian s log ( n ) (non-unif.) [ours] s · PolyLog( n, s ) √ sn · log( n ) (unif.) [CX15] [CX15; NW13a; NW13b] Fourier s · PolyLog( n, s ) [CRT06; KW14; Poo15] Table 1: An overview of known asymptotic-order sampling ratesfor TV minimization in compressed sensing, ignoring universaland model-dependent constants.
The main contribution of this work consists in breakingthe aforementioned √ sn -complexity barrier. Taking a non-uniform, signal-dependent perspective, we show that a largeclass of gradient-sparse signals is already recoverable from m (cid:38) s · PolyLog( n ) Gaussian measurements. Note that sucha result does not contradict the findings of Cai and Xu [CX15],as these are formulated uniformly across all s-gradient-sparse.Indeed, the √ sn -rate describes the worst-case performance onthe class of all s -gradient-sparse signals. We show that a mean-ingful restriction of this class allows for a significant improve-ment of the situation, cf. the numerical experiments of [CX15;GKM20]. With that in mind, our analysis reveals that the sepa-ration distance of jump discontinuities of x ∗ is crucial: Definition 3.1 (Separation constant) Let x ∗ ∈ R n be a sig-nal with s > jump discontinuities such that supp( ∇ x ∗ ) = { ν , . . . , ν s } where ν < ν < · · · < ν s < ν s +1 := n .We say that x ∗ is ∆ -separated for some separation constant ∆ > if min i ∈ [ s +1] | ν i − ν i − | n ≥ ∆ s + 1 . It is not hard to see that the separation constant can always bechosen such that ( s + 1) /n ≤ ∆ ≤ , where larger values of ∆ indicate that the gradient support is closer to being equidistant.Indeed, in the (optimal) case of equidistantly distributed singu-larities, ∆ = 1 is a valid choice, independently of s . Based onthis notion of separation, our main result reads as follows: Theorem 3.2 (Exact recovery via TV minimization)
Let x ∗ ∈ R n be a ∆ -separated signal with s > jump discontinuitiesand ∆ ≥ s/n . Let u > and assume that A ∈ R m × n is astandard Gaussian random matrix with m (cid:38) ∆ − · s log ( n ) + u . Then with probability at least − e − u / , TV minimization (TV-1) with input y = Ax ∗ ∈ R m recovers x ∗ exactly. The proof of Theorem 3.2 relies on a sophisticated upperbound for the associated conic Gaussian mean width, which isbased on a signal-dependent, non-dyadic Haar wavelet trans-form. As such, the latter result can be extended to sub-Gaussian measurements as well as stable and robust recovery;see [GMS20, Sec. 2.4] for more details.The significance of Theorem 3.2 depends on the size ofthe separation constant ∆ . In particular, we obtain the near-optimal rate of m (cid:38) s · PolyLog( n ) if ∆ can be chosen in- dependently of n and s . A typical example of such a situa-tion is the discretization of a suitable piecewise constant func-tion X : (0 , → R . Indeed, based on Theorem 3.2, [GMS20,Cor. 2.6] shows that m (cid:38) s · log ( n ) measurements are suffi-cient for exact recovery when X is finely enough discretized;see Figure 1 for a visualization of this result. (a) (b)(c) (d)Figure 1: Numerical simulation.
Subfigure (a) and (b) showschematic examples of the signal classes that are considered inthis experiment at different resolution levels. The orange signal(with circle symbols) is defined as discretization of the piece-wise constant function X : (0 , → R with s = 5 jump dis-continuities that is plotted in black. The blue plot (with di-amond symbols) shows a so-called dense-jump signal , whichdoes not match the intuitive notion of a -gradient-sparse sig-nal; note that the spatial location of the jumps is chosen adap-tively to the resolution level here, which does not correspond toa discretization of a piecewise constant function. For each sig-nal class we have created phase transition plots: Subfigure (c)and (d) display the empirical probability of successful recoveryvia TV minimization (TV-1) for different pairs of ambient di-mension n and number of measurements m ; note the horizon-tal axis uses a logarithmic scale. The corresponding grey tonesreflect the observed probability of success, reaching from cer-tain failure (black) to certain success (white). Additionally, wehave estimated the conic Gaussian mean width of (cid:107)∇ ( · ) (cid:107) at x ∗ (denoted by w ∧ ( D ( (cid:107)∇ ( · ) (cid:107) , x ∗ )) ), which is known to preciselycapture the phase transition (cf. [ALMT14]). The result of Sub-figure (d) confirms that the class of dense-jump signals suffersfrom the √ sn -bottleneck as predicted by [CX15]. On the otherhand, Subfigure (c) reveals that this bottleneck can be broken fordiscretized signals, as predicted by Theorem 3.2. We have shown that the √ sn -bottleneck for 1D TV recoveryfrom Gaussian measurement can be broken for signals withwell separated jump discontinuities. The results of Table 1suggest that TV minimization in one spatial dimension playsa special role in this regard. However, we argue that sucha phenomenon can also be observed in higher spatial dimen-sions. In fact, we conjecture that the common rate of m (cid:38) s · PolyLog( n, s ) only reflects worst-case scenarios, while itcan be significantly improved for natural signal classes, such aspiecewise constant functions with sufficiently smooth bound-aries. eferences [ALMT14] D. Amelunxen, M. Lotz, M. B. McCoy, and J. A. Tropp.“Living on the edge: phase transitions in convex programswith random data”. Inf. Inference
Con-fluentes Math.
Inf. Inference
IEEE Trans. Inf. Theory
J. Math. Imaging Vis.
Numer. Math.
IEEE Signal Proc. Lett. (cid:96) -Analysis Min-imization and Generalized (Co-)Sparsity: When Does Re-covery Succeed?” Appl. Comput. Harmon. Anal. (2020), on-line, DOI: https://doi.org/10.1016/j.acha.2020.01.002 .[Gra07] M. Grasmair. “The Equivalence of the Taut String Algo-rithm and BV-Regularization”.
J. Math. Imaging Vis.
IEEE Trans. Imag.Proc.
Com-pressed Sensing and its Applications: Second InternationalMATHEON Conference 2015 . Ed. by H. Boche, G. Caire,R. Calderbank, M. M¨arz, G. Kutyniok, and R. Mathar.Birkh¨auser, 2017, 333–358.[LJ10] M. A. Little and N. S. Jones. “Sparse Bayesian step-filteringfor high-throughput analysis of molecular machine dynam-ics”.
IEEE International Conference on Acoustics, Speech,and Signal Processing (ICASSP 2010) . 2010, 4162–4165.[LJ11] M. A. Little and N. S. Jones. “Generalized methods andsolvers for noise removal from piecewise constant signals.I. Background theory”.
Proc. Royal Soc. Lond. A
Ann. Statist.
IEEETrans. Imag. Proc.
SIAM J. Imag. Sci.
IEEE Trans. Pattern Anal.Mach. Intell.
SIAM J. Imag. Sci.
Physica D: NonlinearPhenomena
SIAM J. Appl. Math.
IEEE Signal Process. Lett.
Total variation denoising (an MM algo-rithm) . Connexions. Dec. 2012.[WWL14] X. Wu, Q. Wang, and M. Liu. “In-situ Soil Moisture Sens-ing: Measurement Scheduling and Estimation Using SparseSampling”.