On the optimality of Orthogonal Greedy Algorithm for M-coherent dictionaries
aa r X i v : . [ m a t h . NA ] M a r On the optimality of Orthogonal Greedy Algorithm for M -coherent dictionaries. Eugene Livshitz
Abstract
We show that Orthogonal Greedy Algorithms (Orthogonal Matching Pursuit)provides almost optimal approximation on the first [1 / (20 M )] steps for M -coherentdictionaries. In this article we continue the research of convergence of greedy algorithms with regardsto dictionaries with small coherence (see [3], [4], [7], [1], [2], [6], [5]). The study of approx-imation by incoherent dictionaries was mainly motivated by applications to compressedsensing. In [3], [7], [1] it was shown that Orthogonal Greedy Algorithm (OrthogonalMatching Pursuit) is effective for signal recovering. In this article we discuss this problemfrom the point of view of Approximation Theory.Let us recall standard definition of Greedy Algorithms theory. We say that a set D from a Hilbert space H is a dictionary if φ ∈ D ⇒ k φ k = 1 , and span D = H. We study dictionaries with small values of coherence M := sup φ,ψ ∈D , φ = ψ |h φ, ψ i| . (1)Dictionaries with coherence M are called M -coherent. Orthogonal Greedy Algorithm (OGA)
Set f := f ∈ H , G OGA ( f, D ) := 0 .For each m ≥ we inductively find g m +1 ∈ D such that |h f m , g m +1 i| = sup g ∈D |h f m , g i| and define G OGAm +1 ( f, D ) := Proj span( g ,...,g m +1 ) ( f ) ,f m +1 := f − G OGAm +1 ( f, D ) . For a function f ∈ H we define its best m -term approximation σ m ( f ) := σ m ( f, D ) := inf c i ∈ R ,φ i ∈D , ≤ i ≤ m k f − m X i =1 c i φ i k . Following V.N. Temlyakov we call inequalities connecting the error of Greedy approx-imation and the best m -term approximation Lebesgue type inequalities . This research is partially supported by Russian Foundation for Basic Research project 08-01-00799and 09-01-12173
Theorem A.
For every M -coherent dictionary D and any function f ∈ H the in-equality k f − G OGAm ( f, D ) k = k f m k ≤ m / σ m ( f ) holds for all m , ≤ m ≤ √ M − . This estimate was improved by J. Tropp [7] (see also paper [1] of D. L. Donoho, M. Eladand V.N. Temlyakov.)
Theorem B.
For every M -coherent dictionary D and any function f ∈ H k f m k ≤ (1 + 6 m ) / σ m ( f ) , if ≤ m ≤ M .
D. L. Donoho, M. Elad and V.N. Temlyakov [2] dramatically improved factor in frontof σ . Theorem C.
For every M -coherent dictionary D and any function f ∈ H k f ⌊ m log m ⌋ k ≤ σ m ( f ) , if ≤ m ≤ M / . V.N. Temlyakov and P. Zheltov [6] improved the upper border for m and proved twonew Lebesgue type inequalities. Theorem D.
For every M -coherent dictionary D and any function f ∈ H k f m ⌊ √ log m ⌋ k ≤ σ m ( f ) , if m √ m ≤ M .
Theorem E.
For every M -coherent dictionary D , any function f ∈ H and any fixed δ > k f m ⌈ δ ⌉ k ≤ σ m ( f ) , if m ≤ (cid:18) M (cid:19) δ −⌈ δ ⌉ . The aim of this article is to prove the following result.
Theorem 1.
For every M -coherent dictionary D and any function f ∈ H we have k f − G OGA m ( f, D ) k = k f m k ≤ σ m ( f ) for all ≤ m ≤ M . Preliminary lemmas.
By conditions of Theorem 1 we have M ≤ mM ≤ / . (2)We use several standard lemmas. Lemma 1.
For any n , ≤ n ≤ m , and h = n X i =1 c i φ i , c i ∈ R , φ i ∈ D , we have max ≤ i ≤ n |h h, φ i i| ≤ max ≤ i ≤ n | c i | (1 + 2 mM ) , (3)max ≤ i ≤ n |h h, φ i i| ≥ max ≤ i ≤ n | c i | (1 − mM ) , (4)max ≤ i ≤ n | c i | ≤ max ≤ i ≤ n |h h, φ i i| (1 + 3 mM ) , (5) Proof.
Using (1) we have for any 1 ≤ i ≤ n h h, φ i i = h c i φ i , φ i i + h X ≤ j ≤ n, i = j c j φ j , φ i i ≤≤ c i + ( n − (cid:18) max ≤ i ≤ n | c i | (cid:19) M ≤ c i + (cid:18) max ≤ i ≤ n | c i | (cid:19) mM. Similarly h h, φ i i ≥ c i − (cid:18) max ≤ i ≤ n | c i | (cid:19) mM. The last two inequalities imply (3) and (4). It follow form (2) that(1 − mM )(1 + 3 mM ) = 1 + mM − mM ) ≥ mM (1 − . ≥ . To prove (5) we estimatemax ≤ i ≤ n | c i | ≤ max ≤ i ≤ n |h h, φ i i| (1 − mM ) − ≤ max ≤ i ≤ n |h h, φ i i| (1 + 3 mM ) . As corollary we obtain
Lemma 2.
Let n ≤ m , h ∈ H , φ i ∈ D , ≤ i ≤ n . Suppose that Proj span( φ ,...,φ n ) ( h ) = n X i =1 c i φ i . Then max ≤ i ≤ n | c i | ≤ max ≤ i ≤ n |h h, φ i i| (1 + 3 mM ) , roof. Set h ′ = Proj span( φ ,...,φ n ) ( h ) . It’s clear that h h, φ i i = h h ′ , φ i i , ≤ i ≤ n. Thus the lemma follows from inequality (5) for h ′ .For n ≥ d n := h f n − , g n i . (6)Let numbers x i,n , n ≥
1, 1 ≤ i ≤ n , satisfy the equality f n = f n − − n X i =1 x i,n g i . (7) Lemma 3.
For any n ≤ m we have | x i,n | ≤ M | d n | (1 + 3 mM ) , ≤ i ≤ n − , (8) | x n,n − d n | ≤ M | d n | (1 + 3 mM ) . (9) Proof.
By definition of OGA h f l , g i i = 0 , ≤ i ≤ l, (10)and f n = f − G OGAn ( f, D ) = f − Proj span( g ,...,g n ) ( f ) . Hence f n = f n − − Proj span( g ,...,g n ) ( f n − ) = f n − − d n g n − Proj span( g ,...,g n ) ( f n − − d n g n ) . (11)Using (1) and (10) we have for h := f n − − d n g n and 1 ≤ i ≤ n − |h h, g i i| ≤ |h f n − , g i i| + | d n h g i , g n i| ≤ M | d n | , h h, g n i = 0 . Suppose that x ′ i,n , 1 ≤ i ≤ n satisfyProj span( g ,...,g n ) ( f n − − d n g n ) = Proj span( g ,...,g n ) ( h ) = n X i =1 x ′ i,n g i . By Lemma 2 | x ′ i,n | ≤ M | d n | (1 + 3 mM ) , ≤ i ≤ n (12)It follows from (11) that f n = f n − − d n g n − n X i =1 x ′ i,n g i = f n − − n X i =1 x i,n g i , where x i,n = x ′ i,n , 1 ≤ i ≤ n − x n,n = d n + x ′ n,n . This and (12) complete theproof. 4 emma 4. For ≤ n ≤ m − we have | d n +1 | ≤ | d n | (1 + 1 . M ) . Proof.
By definition of OGA |h f n − , g n +1 i| ≤ |h f n − , g n i| = | d n | , Using Lemma 3, (1) and (2) we have |h f n , g n +1 i| ≤ |h f n − − n X i =1 x i,n g i , g n +1 i| ≤ |h f n − , g n +1 i| + n X i =1 | x i,n h g i , g n +1 i| ≤≤ | d n | + M | x n,n | + n − X i =1 | x i,n | ! ≤ | d n | + ( nM | d n | (1 + 3 mM ) + | d n | ) M ≤≤ | d n | (1 + (2 mM (1 + 3 mM ) + 1) M ) ≤ | d n | (1 + 1 . M ) . Lemma 5.
For any ≤ l ≤ n ≤ m we have | d n | ≤ | d l | exp(2 . mM ) . Proof.
Using Lemma 4 we write | d n | ≤ | d l | (1 + 1 . M ) n − l ≤ | d l | (cid:18) . mM m (cid:19) m ≤ | d l | exp(2 . mM ) . By the definition of the best m -term approximation there exist a j ∈ R , ψ j ∈ D , 1 ≤ j ≤ m ,and v ∈ H such that f = f = m X j =1 a j ψ j + v , h v , ψ j i = 0 , ≤ j ≤ m, (13) k v k ≤ . σ m ( f ) . (14)Set L := span( ψ , . . . , ψ m ) , P L ( · ) := Proj L ( · ) , P ⊥ L ( · ) := Proj L ⊥ ( · ) ,v n := P ⊥ L ( f n ) , ≤ n ≤ m. Let numbers a j,n and b j,n , 0 ≤ n ≤ m , 1 ≤ j ≤ m satisfy equalities f n = P L ( f n ) + P ⊥ L ( f n ) = m X j =1 a j,n ψ j + v n . (15)5 X j =1 b j,n ψ j = P L ( f − f n ) . (16)Then a j,n = a j − b j,n , ≤ j ≤ m, ≤ n ≤ m. (17)Define T := (cid:8) i ∈ { , . . . , m } : g i ∈ { ψ j } mj =1 (cid:9) .T := { , . . . , m } \ T ,S := (cid:8) j ∈ { , . . . , m } : ψ j ∈ { g n } mn =1 (cid:9) , (18) S := { , . . . , m } \ S . (19)For numbers x i,n , 1 ≤ i ≤ n ≤ m , from (7) and for d n from (6) we define x n := X ≤ i ≤ n, i ∈ T | x i,n | ,D := X ≤ n ≤ m, n ∈ T d n . Lemma 6.
Let ≤ i < n ≤ m , i, n ∈ T . Then we have |h P ⊥ L ( g n ) , g i i| ≤ . M Proof.
Let P L ( g n ) = m X j =1 c j ψ j . Since n ∈ T and g n = ψ j , |h g n , ψ j i| ≤ M, ≤ j ≤ m, we get by Lemma 2 that max ≤ j ≤ m | c j | ≤ M (1 + 3 mM ) . Therefore we have |h P ⊥ L ( g n ) , g i i| = |h g n − P L ( g n ) , g i i| ≤ |h g n , g i i| + |h P L ( g n ) , g i i| ≤≤ M + |h m X j =1 c j ψ j , g i i| ≤ M + m (cid:18) max ≤ j ≤ m | c j | (cid:19) max ≤ j ≤ m |h ψ j , g i i| ≤≤ M + ( mM ) M (1 + 3 mM ) ≤ . M. emma 7. Let n ∈ T then x n ≤ . D / m − / , k v n k ≤ k v n − k + 0 . DM.
Proof.
Let u n := ♯ ( T ∩ { , . . . , n } ) . If T ∩ { , . . . , n } = ∅ then x n = 0, v n = v n − = v and nothing to prove, so we mayassume that u n ≥
1. By Lemma 5 | d n | ≤ exp(2 . mM ) min ≤ i ≤ n, i ∈ T | d i | . (20)On the other hand we have (cid:18) min ≤ i ≤ n, i ∈ T | d i | (cid:19) u n ≤ X ≤ i ≤ n, i ∈ T d i ≤ X ≤ i ≤ m, i ∈ T d i = D. Combining with (20) we obtain | d n | ≤ exp(2 . mM ) (cid:18) Du n (cid:19) / . (21) d n u n ≤ exp(5 mM ) D. (22)Applying Lemma 3, (2) and (21) we write x n = X ≤ i ≤ n, i ∈ T | x i,n | = X ≤ i ≤ n − , i ∈ T | x i,n | ≤ M | d n | (1 + 3 mM ) u n ≤≤ M (1 + 3 mM ) exp(2 . mM )( Du n ) / ≤ M (1 + 3 mM ) exp(2 . mM )( D m ) / == (2 D ) / M m / (1 + 3 mM ) exp(2 . mM ) ≤ . D / m − / . (23)We have that for any l , 1 ≤ l ≤ m we have |h m X j =1 a j,n − ψ j , ψ l i| = |h m X j =1 a j,n − ψ j + v n − , ψ l i| ≤ |h f n − , ψ l i| ≤ | d n | . Then by Lemma 1 we getmax ≤ j ≤ m | a j,n − | ≤ max ≤ l ≤ m |h m X j =1 a j,n − ψ j , ψ l i| ! (1 + 3 mM ) ≤ | d n | (1 + 3 mM ) . (24)Define h := X ≤ i ≤ n,i ∈ T x i,n g i = X ≤ i ≤ n − ,i ∈ T x i,n g i . (25)According the definition of v n we have v n = P ⊥ L ( f n ) = P ⊥ L f n − − n X i =1 x i,n g i ! = v n − − P ⊥ L ( h ) , v n k = k v n − − P ⊥ L ( h ) k ≤ k v n − k + 2 |h v n − , P ⊥ L ( h ) i| + k P ⊥ L ( h ) k ≤≤ k v n − k + 2 |h v n − , h i| + k h k . (26)By definition of OGA h f n − , g i i = 0, 1 ≤ i ≤ n −
1, therefore using (25) and (15) h f n − , h i = 0 , |h v n − , h i| = |h f n − − m X j =1 a j,n − ψ j , h i| = m X j =1 |h a j,n − ψ i , h i| ≤≤ m X j =1 | a j,n − | X ≤ i ≤ n − , i ∈ T |h ψ j , x i,n g i i| . Applying (1), (22), (24) and Lemma 3 we obtain |h v n − , h i| ≤ m X j =1 | a j,n − | X ≤ i ≤ n − , i ∈ T | x i,n h ψ j , g i i| ≤≤ | d n | (1 + 3 mM ) M | d n | (1 + 3 mM ) u n mM ≤≤ ( d n u n )(1 + 3 mM ) mM ≤ D exp(5 mM )(1 + 3 mM ) mM . k h k = X ≤ i ≤ n, i ∈ T x i,n h g i , g i i + 2 X ≤ i,l ≤ n, i,l ∈ T , i = l x i,n x l,n h g i , g l i ≤≤ M | d n | (1 + 3 mM ) u n + 2 M | d n | (1 + 3 mM ) u n M ≤≤ M | d n | u n (1 + 3 mM ) + M | d n | u n (1 + 3 mM ) mM ≤≤ DM exp(5 mM )(1 + 3 mM ) (1 + 4 mM ) . From (26) and (2) it follows that k v n k ≤ k v n − k + 2 |h v n − , h i| + k h k ≤≤ k v n − k + DM exp(5 mM )(1 + 3 mM ) (2 mM + M + 4 mM ) ≤≤ k v n − k + DM exp(5 mM )(1 + 3 mM ) (3 mM + 4( mM ) ) ≤ k v n − k + 0 . DM.
This estimate together with (23) proves the lemma.
Lemma 8.
Let n ∈ T then x n ≤ . | d n | , k v n k ≤ k v n − k − . d n Proof.
Applying Lemma 3 we have x n = X ≤ i ≤ n, i ∈ T | x i,n | ≤ d n + X ≤ i ≤ n, i ∈ T M | d n | (1 + 3 mM ) ≤≤ | d n | (1 + 2 mM (1 + 3 mM )) ≤ | d n | (1 + 3 mM ) ≤ . | d n | .
8y Lemma 1 we havemax ≤ j ≤ m | a j,n − | ≤ (1 + 3 mM ) max ≤ l ≤ m (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) h m X j =1 a j,n − ψ j , ψ l i (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) == (1 + 3 mM ) max ≤ l ≤ m (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) h m X j =1 a j,n − ψ j + v n − , ψ l i (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) == (1 + 3 mM ) max ≤ l ≤ m |h f n − , ψ l i| ≤ (1 + 3 mM ) | d n | . (27)Therefore |h v n − , g n i − d n | = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) h f n − − m X j =1 a j,n − ψ j , g n i − d n (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) == (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) h f n − , g n i − m X j =1 a j,n − h ψ j , g n i − d n (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) m X j =1 a j,n − h ψ j , g n i (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤≤ (cid:18) max ≤ j ≤ m | a j,n − | (cid:19) m max ≤ j ≤ m |h ψ j , g n i| ≤ (1 + 3 mM ) | d n | mM. (28)Set v ′ n := P ⊥ L ( f n − − x n,n g n ) . Using Lemma 3, (2) and (28) we estimate2 x n,n h v n − , g n i = 2( d n + ( x n,n − d n ))( d n + ( h v n − , g n i − d n )) ≥≥ | d n | − M (1 + 3 mM ) | d n | )( | d n | − (1 + 3 mM ) | d n | mM ) ≥≥ d n − | d n | (1 + 3 mM )( M + mM ) ≥ d n − | d n | (1 + 3 mM ) mM, k v ′ n k ≤ k P ⊥ L ( f n − − x n,n g n ) k = k v n − − x n,n P ⊥ L ( g n ) k ≤≤ k v n − k − x n,n h v n − , P ⊥ L ( g n ) i + x n,n k P ⊥ L ( g n ) k ≤≤ k v n − k − x n,n h v n − , g n i + x n,n ≤≤ k v n − k − d n + 4 | d n | (1 + 3 mM ) mM + ( | d n | + M | d n | (1 + 3 mM )) ≤≤ k v n − k − . d n . (29)Similar to the proof of Lemma 7 we define h := X ≤ i ≤ n − ,i ∈ T x i,n g i . Equalities h f n − , g i i = 0, 1 ≤ i ≤ n − h f n − , h i = 0 . |h v ′ n , h i| = |h P ⊥ L ( f n − ) − x n,n P ⊥ L ( g n ) , h i| = |h v n − − x n,n P ⊥ L ( g n ) , h i| == |h f n − − m X j =1 a j,n − ψ j − x n,n P ⊥ L ( g n ) , h i| ≤ m X j =1 |h a j,n − ψ i , h i| + | x n,n h P ⊥ L ( g n ) , h i| ≤≤ m X j =1 | a j,n − | X ≤ i ≤ n − , i ∈ T |h ψ j , x i,n g i i| + X ≤ i ≤ n − , i ∈ T | x n,n x i,n h P ⊥ L ( g n ) , g i i| . Applying (1), (27), Lemma 3 and Lemma 6 we continue |h v ′ n , h i| ≤ m X j =1 | a j,n − | X ≤ i ≤ n − , i ∈ T | x i,n ||h ψ j , g i i| + X ≤ i ≤ n − , i ∈ T | x n,n x i,n h P ⊥ L ( g n ) , g i i| ≤≤ max ≤ j ≤ m | a j,n − | max ≤ i ≤ n − , i ∈ T | x i,n | m X j =1 X ≤ i ≤ n − , i ∈ T M ++ | d n | (1 + M (1 + 3 mM )) max ≤ i ≤ n − , i ∈ T | x i,n | X ≤ i ≤ n − , i ∈ T . M ≤≤ d n (1 + 3 mM ) nmM + 1 . d n (1 + M (1 + 3 mM ))(1 + 3 mM ) nM ≤≤ d n (1 . mM ) + 1 . d n (1 + mM (1 . . mM ≤ . d n , (30) k h k = X ≤ i ≤ n − , i ∈ T x i,n h g i , g i i + 2 X ≤ i,l ≤ n − , i,l ∈ T , i = l x i,n x l,n h g i , g l i ≤≤ M | d n | (1 + 3 mM ) n + 2 M | d n | (1 + 3 mM ) n M ≤≤ d n (1 + 3 mM ) M (2 M m + 8(
M m ) ) ≤ . d n . (31)Using definitions of v ′ n and h we write v n = P ⊥ L ( f n ) = P ⊥ L f n − − n X i =1 x i,n g i ! = P ⊥ L ( f n − − x n,n g n ) − P ⊥ L n − X i =1 x i,n g i ! == v ′ n − P ⊥ L X ≤ i ≤ n − , i ∈ T x i,n g i ! = v ′ n − P ⊥ L ( h ) , k v n k = k v ′ n k − h v ′ n , P ⊥ L ( h ) i + k P ⊥ L ( h ) k ≤ k v ′ n k + 2 |h v ′ n , h i| + k h k . Applying (29), (30) and (31) we obtain k v n k ≤ k v ′ n k + 2 |h v ′ n , P ⊥ L ( h ) i| + k h k ≤≤ k v n − k − . d n + 2(0 . d n ) + 0 . d n ≤ k v n − k − . d n emma 9. We have m X n =1 x n ≤ D / m / . Proof.
Using Cauchy inequality, Lemma 8 and Lemma 7 we get X ≤ n ≤ m, n ∈ T x n ≤ X ≤ n ≤ m, n ∈ T . | d n | ≤≤ . X ≤ n ≤ m, n ∈ T d n ! / (2 m ) / ≤ . D / m / , m X n =1 x l = X ≤ n ≤ m, n ∈ T x n + X ≤ n ≤ m, n ∈ T x n ≤ ♯T max ≤ n ≤ m, n ∈ T x n + X ≤ n ≤ m, n ∈ T x n ≤≤ m (0 . D / m − / ) + 1 . D / m / ≤ D / m / . Lemma 10.
We have D / ≤ . σ m ( f ) , k v m k ≤ k v k . Proof.
Applying (14), Lemma 7 and Lemma 8 we write(1 . σ m ( f )) ≥ k v k ≥ k v k − k v m k = m X n =1 ( k v n − k − k v n k ) == X ≤ n ≤ m, n ∈ T ( k v n − k − k v n k ) + X ≤ n ≤ m, n ∈ T ( k v n − k − k v n k ) ≥≥ ♯T ( − . DM ) + X ≤ n ≤ m, n ∈ T . d n ≥ m ( − . DM ) + 0 . D ≥ . D. Hence D / ≤ . . − / σ m ( f ) ≤ . σ m ( f ) . In the next lemma we use definitions (16), (18) and (19).
Lemma 11.
For any ≤ n ≤ m and j ∈ S | b j,n | ≤ . D / m − / . roof. By definition (7) f n − f = n X l =1 l X i =1 x i,l g i = n X i =1 g i n X l = i x i,l ! == X ≤ i ≤ n, i ∈ T g i n X l = i x i,l ! + X ≤ i ≤ n, i ∈ T g i n X l = i x i,l ! . Assume that numbers b b j,n and e b j,n , 1 ≤ j ≤ m , 1 ≤ n ≤ m satisfy m X j =1 b b j,n ψ j = P L X ≤ i ≤ n, i ∈ T g i n X l = i x i,l !! , m X j =1 e b j,n ψ j = P L X ≤ i ≤ n, i ∈ T g i n X l = i x i,l !! . It follows from (16) that b j,n = b b j,n + e b j,n , ≤ j ≤ m, ≤ n ≤ m. (32)It’s clear that P L X ≤ i ≤ n, i ∈ T g i n X l = i x i,l !! = X ≤ i ≤ n, i ∈ T g i n X l = i x i,l ! and therefore b b j,n = 0 , j ∈ S , ≤ n ≤ m. (33)Set h = X ≤ i ≤ n, i ∈ T g i n X l = i x i,l ! . By Lemma 9 we estimate for each j , 1 ≤ j ≤ m , |h h, ψ j i| ≤ X ≤ i ≤ n, i ∈ T h g i , ψ j i n X l = i | x i,l | ! ≤≤ M n X l =1 X ≤ i ≤ l, i ∈ T | x i,l | ≤ M n X l =1 x l ≤ M m X l =1 x l ≤ D / m / M. According Lemma 2 we have for 1 ≤ j ≤ m and 1 ≤ n ≤ m | e b j,n | ≤ D / m / M (1 + 3 mM ) . (34)Combining (2), (32), (33) and (34) we obtain | b j,n | ≤ D / m / M (1 + 3 mM ) ≤ . D / m − / , j ∈ S , ≤ n ≤ m. Proof of Theorem 1.
First we estimate | a j, m | , j ∈ S . For each 1 ≤ n ≤ m by Lemma 1 and Lemma 11 wehave | d n | ≥ max ≤ j ≤ m |h f n − , ψ j i| = max ≤ j ≤ m |h P L ( f n − ) , ψ j i| ≥ (1 − mM ) max ≤ j ≤ m | a j,n − | ≥≥ (1 − mM ) max j ∈ S | a j,n − | ≥ (1 − mM ) max j ∈ S | a j − b j,n − | ≥≥ (1 − mM ) (cid:18) max j ∈ S | a j | − . D / m − / (cid:19) . Since ♯T ≤ m and ♯T ≥ m we get D = X ≤ i ≤ m, i ∈ T d n ≥ m (cid:18) (1 − mM ) (cid:18) max j ∈ S | a j | − . D / m − / (cid:19)(cid:19) . Hence (cid:18) max j ∈ S | a j | − . D / m − / (cid:19) ≤ D / m − / (1 + 3 mM ) , max j ∈ S | a j | ≤ . D / m − / + 0 . D / m − / = 1 . D / m − / . Then by (17) and Lemma 11 for any j ∈ S we obtain | a j, m | = | a j − b j, m | ≤ . D / m − / + 0 . D / m − / ≤ . D / m − / . We use well known inequality (see, for example, Lemma 2.1 from [2]) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)X j ∈ S a j, m ψ j (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ X j ∈ S a j, m ! (1 + mM ) ≤ m (cid:0) . D / m − / (cid:1) (1 . ≤ . D. Using the definition of OGA, (15), Lemma 10 and (14) we estimate k f m k = min c i , ≤ i ≤ m k f m − m X i =1 c i g i k = min c i , ≤ i ≤ m k m X j =1 a j, m ψ j + v m − m X i =1 c i g i k ≤≤ min c i , ≤ i ≤ m k m X j =1 a j, m ψ j − m X i =1 c i g i k + k v m k ≤≤ min c l , l ∈ S k m X j =1 a j, m ψ j − X l ∈ S c l ψ l k + k v k ≤ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)X j ∈ S a j, m ψ j (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + k v k ≤≤ (2 . D ) / + 1 . σ m ( f ) ≤ (2 . / . σ m ( f ) + 1 . σ m ( f ) ≤ σ m ( f ) . This completes the proof. (cid:3)
The author is grateful professor V.N. Temlyakov and professor S.V. Konyagin foruseful discussions. 13 eferences [1] Donoho D. L., Elad M., Temlyakov V. N. “ Stable recovey of sparse overcompleterepresentations in the presense of noise” //
IEEE Trans. Inform. Th.
Journal of Approximation Theory
Proc. 14th Annu. ACM-SIAM Symp. DiscreteAlgorithms.
Proc. Int Conf. Independent Component Anal. (ICA’04)