Convergence Analysis of Projection Method for Variational Inequalities
Yekini Shehu, Olaniyi. S. Iyiola, Xiao-Huan Li, Qiao-Li Dong
aa r X i v : . [ m a t h . O C ] J a n Convergence Analysis of Pro jection Method forVariational Inequalities
Yekini Shehu ∗ Olaniyi. S. Iyiola † Xiao-Huan Li ‡ and Qiao-Li Dong § January 22, 2021
Abstract
The main contributions of this paper are the proposition and the convergenceanalysis of a class of inertial projection-type algorithm for solving variationalinequality problems in real Hilbert spaces where the underline operator ismonotone and uniformly continuous. We carry out a unified analysis of theproposed method under very mild assumptions. In particular, weak conver-gence of the generated sequence is established and nonasymptotic O (1 /n ) rateof convergence is established, where n denotes the iteration counter. We alsopresent some experimental results to illustrate the profits gained by introduc-ing the inertial extrapolation steps. We first state the formal definition of some classes of functions that play an essentialrole in this paper.Let H be a real Hilbert space and X ⊆ H be a nonempty subset. Definition 1.1.
A mapping F : X → H is called (a) monotone on X if h F ( x ) − F ( y ) , x − y i ≥ for all x, y ∈ X ; (b) Lipschitz continuous on X if there exists a constant L > such that k F ( x ) − F ( y ) k ≤ L k x − y k , ∀ x, y ∈ X. ∗ Department of Mathematics, Zhejiang Normal University, Jinhua, 321004, People’s Republicof China; Institute of Science and Technology (IST), Am Campus 1, 3400, Klosterneuburg, Vienna,Austria; e-mail: [email protected]. † Department of Mathematics, Minnesota State University-Moorhead, Minnesota, USA; e-mail:[email protected] ‡ College of Science, Civil Aviation University of China, Tianjin 300300, China.; e-mail: [email protected]. § College of Science, Civil Aviation University of China, Tianjin 300300, China.; e-mail:[email protected]. if for each sequence { x n } we have: { x n } con-verges weakly to x implies { F ( x n ) } converges weakly to F ( x ) . Let C be a nonempty, closed and convex subset of H and F : C → H be a continuousmapping. The variational inequality problem (for short, VI( F, C )) is defined as: find x ∈ C such that h F ( x ) , y − x i ≥ , ∀ y ∈ C. (1)Let SOL denote the solution set of VI( F, C ) (1). Variational inequality theory is animportant tool in economics, engineering mechanics, mathematical programming,transportation, and so on (see, for example, [7, 8, 22, 29–31, 38]).A well-known projection-type method for solving VI(
F, C ) (1) is the extragradientmethod introduced by Korpelevich in [32]. It is well known that the extragradientmethod requires two projections onto the set C and two evaluations of F per itera-tion.One important hallmark in the design of numerical methods related to the extragra-dient method is to minimize the number of evaluations of P C per iteration becauseif C is a general closed and convex set, then a minimal distance problem has to besolved (twice) in order to obtain the next iterate. This has the capacity to seriouslyaffect the efficiency of the extragradient method in a situation, where a projectiononto C is hard to evaluate and therefore computationally costly.An attempt in this direction was initiated by Censor et al. [18], who modified ex-tragradient method by replacing the second projection onto the closed and convexsubset C with the one onto a subgradient half-space. Their method, which thereforeuses only one projection onto C , is called the subgradient extragradient method: x ∈ H , y n = P C ( x n − λF ( x n )) ,T n := { w ∈ H : h x n − λF ( x n ) − y n , w − y n i ≤ } ,x n +1 = P T n ( x n − λF ( y n )) . (2)Using (2), Censor et al. [18] proved weak convergence result for VI( F, C ) (1) witha monotone and L -Lipschitz-continuous mapping F where λ ∈ (0 , L ). Several otherrelated methods to extragradient method and (2) for solving VI( F, C ) (1) in realHilbert spaces when F is monotone and L -Lipschitz-continuous mapping have beenstudied in the literature (see, for example, [15–17, 21, 26, 35–37, 40, 46]).Motivated the result of Alvarez and Attouch in [2] and Censor et al. in [18], Thongand Hieu [45] introduced an algorithm which is a combination of (2) and inertialmethod for solving VI( F, C ) (1) in real Hilbert space: x , x ∈ H , w n = x n + α n ( x n − x n − ) ,y n = P C ( w n − λF ( w n )) ,T n := { w ∈ H : h w n − λF ( w n ) − y n , w − y n i ≤ } ,x n +1 = P T n ( w n − λF ( y n )) . (3)2hong and Hieu [45] proved that the sequence { x n } generated by (3) convergesweakly to a solution of VI( F, C ) (1) with a monotone and L -Lipschitz-continuousmapping F where 0 < λL ≤ − α − α − δ − α + α for some 0 < δ < − α − α and { α n } is a non-decreasing sequence with 0 ≤ α n ≤ α < √ − F cannot be accurately estimated, and is usually overestimated, thereby resulting intoo small step-sizes. This, of course, is not practical. Therefore, algorithms (2) and(3) are not applicable in most cases of interest. The usual approach to overcomethis difficulty consists in some prediction of a step-size with its further correction(see [29, 38]) or in a usage of an Armijo type line search procedure along a feasibledirection (see [43]). In terms of computations, the latter approach is more effective,since very often the former approach requires too many projections onto the feasibleset per iteration.This paper focuses on the analysis and development of computational projection-type algorithm with inertial extrapolation step for solving VI( F, C ) (1) when theunderline operator F is monotone and uniformly continuous when the feasible set C is a nonempty closed affine subset. We obtain weak convergence of the sequencegenerated by our method. We provide theoretical analysis of our result with weakerassumption on the underline operator F unlike [17,18,35,36] and many other relatedresults on monotone variational inequalities. We also establish the nonasymptotic O (1 /n ) rate of convergence, which is not given before in other previous inertial typeprojection methods for VI( F, C ) (1) (see, e.g., [21, 45]) and give carefully designedcomputational experiments to illustrate our results. Our computational results showthat our proposed methods outperform the iterative methods (2) and (3). Further-more, our result complements some recent results on inertial type algorithms (see,e.g., [2–6, 10, 12, 13, 19, 33, 34, 41, 42]).The paper is organized as follows: We first recall some basic definitions and resultsin Section 2. Some discussions about the proposed inertial projection-type methodare given in Section 3. The weak convergence analysis of our algorithm is theninvestigated in Section 4. We give the rate of convergence of our proposed methodin Section 5 and some numerical experiments can be found in Section 6. We concludewith some final remarks in Section 7.
First, we recall some properties of the projection, cf. [9] for more details. For anypoint u ∈ H , there exists a unique point P C u ∈ C such that k u − P C u k ≤ k u − y k , ∀ y ∈ C. C is called the metric projection of H onto C . We know that P C is a nonexpansivemapping of H onto C . It is also known that P C satisfies h x − y, P C x − P C y i ≥ k P C x − P C y k , ∀ x, y ∈ H. (4)In particular, we get from (4) that h x − y, x − P C y i ≥ k x − P C y k , ∀ x ∈ C, y ∈ H. (5)Furthermore, P C x is characterized by the properties P C x ∈ C and h x − P C x, P C x − y i ≥ , ∀ y ∈ C. (6)Further properties of the metric projection can be found, for example, in Section 3of [23].The following lemmas will be used in our convergence analysis. Lemma 2.1.
The following statements hold in H :(a) k x + y k = k x k + 2 h x, y i + k y k for all x, y ∈ H ;(b) h x − y, x − z i = k x − y k + k x − z k − k y − z k for all x, y, z ∈ H ;(c) k αx + (1 − α ) y k = α k x k + (1 − α ) k y k − α (1 − α ) k x − y k for all x, y ∈ H and α ∈ R . Lemma 2.2. (see [1, Lem. 3]) Let { ψ n } , { δ n } and { α n } be the sequences in [0 , + ∞ ) such that ψ n +1 ≤ ψ n + α n ( ψ n − ψ n − ) + δ n for all n ≥ , P ∞ n =1 δ n < + ∞ and thereexists a real number α with ≤ α n ≤ α < for all n ≥ . Then the following hold: ( i ) P n ≥ [ ψ n − ψ n − ] + < + ∞ , where [ t ] + = max { t, } ;(ii) there exists ψ ∗ ∈ [0 , + ∞ ) such that lim n → + ∞ ψ n = ψ ∗ . Lemma 2.3. (see [9, Lem. 2.39]) Let C be a nonempty set of H and { x n } be asequence in H such that the following two conditions hold:(i) for any x ∈ C , lim n →∞ k x n − x k exists;(ii) every sequential weak cluster point of { x n } is in C .Then { x n } converges weakly to a point in C . The following lemmas were given in R n in [25]. The proof of the lemmas are thesame if given in infinite dimensional real Hilbert spaces. Hence, we state the lemmasand omit the proof in real Hilbert spaces. Lemma 2.4.
Let C be a nonempty closed and convex subset of H . Let h be a real-valued function on H and define K := { x : h ( x ) ≤ } . If K is nonempty and h isLipschitz continuous on C with modulus θ > , then dist( x, K ) ≥ θ − max { h ( x ) , } , ∀ x ∈ C, where dist( x, K ) denotes the distance function from x to K . emma 2.5. Let C be a nonempty closed and convex subset of H , y := P C ( x ) and x ∗ ∈ C . Then k y − x ∗ k ≤ k x − x ∗ k − k x − y k . (7)The following lemma was stated in [28, Prop. 2.11], see also [27, Prop. 4]. Lemma 2.6.
Let H and H be two real Hilbert spaces. Suppose F : H → H isuniformly continuous on bounded subsets of H and M is a bounded subset of H .Then F ( M ) is bounded. Finally, the following result states the equivalence between a primal and a weakform of variational inequality for continuous, monotone operators.
Lemma 2.7. ( [44, Lem. 7.1.7]) Let C be a nonempty, closed, and convex subset of H . Let F : C → H be a continuous, monotone mapping and z ∈ C . Then z ∈ SOL ⇐⇒ h F ( x ) , x − z i ≥ for all x ∈ C. Let us first state the assumptions that we will assume to hold for the rest of thispaper.
Assumption 3.1.
Suppose that the following hold:(a) The feasible set C is a nonempty closed affine subset of the real Hilbert space H .(b) F : C → H is monotone and uniformly continuous on bounded subsets of H .(c) The solution set SOL of VI( F, C ) is nonempty.
Assumption 3.2.
Suppose the real sequence { α n } and constants β, δ, σ > { α n } ⊂ (0 ,
1) with 0 ≤ α n ≤ α n +1 ≤ α < n .(b) δ > α (1+ α )( α + δσ )+ ασδ ( α + δσ ) σ and β < δσα + δσ − α (1 + α ) − ασδ .Let r ( x ) := x − P C ( x − F ( x ))stand for the residual equation.Observe that if we take y = x − F ( x ) in (5), then we have h F ( x ) , r ( x ) i ≥ k r ( x ) k , ∀ x ∈ C. (8)We now introduce our proposed method below.5 lgorithm 1 Inertial Projection-type Method Choose sequence { α n } and σ ∈ (0 ,
1) such that the conditions from Assump-tion 3.2 hold, and take γ ∈ (0 , x = x ∈ H be a given starting point.Set n := 1. Set w n := x n + α n ( x n − x n − ) . Compute z n := P C ( w n − F ( w n )). If r ( w n ) = w n − z n = 0: STOP. Compute y n = w n − γ m n r ( w n ), where m n is the smallest nonnegative integersatisfying h F ( y n ) , r ( w n ) i ≥ σ k r ( w n ) k . (9)Set η n := γ m n . Compute x n +1 = P C n ( w n ) , (10)where C n = { x : h n ( x ) ≤ } and h n ( x ) := h F ( y n ) , x − y n i . (11) Set n ← n + 1 and goto 2 .It is clear that r ( w n ) = 0 implies that we are at a solution of the variational inequal-ity. In our convergence theory, we will implicitly assume that this does not occurafter finitely many iterations, so that Algorithm 1 generates an infinite sequencesatisfying, in particular, r ( w n ) = 0 for all n ∈ N . We will see that this propertyimplies that Algorithm 1 is well defined. Remark 3.3. (a) Algorithm 1 requires, at each iteration, only one projection ontothe feasible set C and another projection onto the half-space C n (see [14] for for-mula for computing projection onto half-space), which is less expensive than theextragradient method especially for the case when computing the projection ontothe feasible set C is a dominating task during iteration.(b) Our Algorithm 1 is much more applicable than (2) and (3) in the sense thatalgorithm (2) and (3) are applicable only for monotone and L -Lipschitz-continuousmapping F . Thus, the L -Lipschitz constant of F or an estimate of it is needed inorder to implement the iterative method (2) but our Algorithm 1 is applicable for amuch more general class of monotone and uniformly continuous mapping F .(c) We observe that the step-size rule in Step 3 involves a couple of evaluationsof F , but these are often much less expensive than projections onto C which wasconsidered in [29, 38]. Furthermore, using the fact that F is continuous and (8), wecan see that Step 3 in Algorithm 1 is well-defined. ♦ Lemma 3.4.
Let the function h n be defined by (11) . Then h n ( w n ) ≥ ση n k w n − z n k . In particular, if w n = z n , then h n ( w n ) > . If x ∗ ∈ SOL, then h n ( x ∗ ) ≤ . roof. Since y n = w n − η n ( w n − z n ), using (9) we have h n ( w n ) = h F ( y n ) , w n − y n i = η n h F ( y n ) , w n − z n i ≥ η n σ k w n − z n k ≥ . If w n = z n , then h n ( w n ) >
0. Furthermore, suppose x ∗ ∈ SOL. Then by Lemma 2.7we have h F ( x ) , x − x ∗ i ≥ x ∈ C. In particular, h F ( y n ) , y n − x ∗ i ≥ h n ( x ∗ ) ≤ . Observe that, in finding η n , the operator F is evaluated (possibly) many times, butno extra projections onto the set C are needed. This is in contrast to a couple ofrelated algorithms for the solution of monotone variational inequalities where thecalculation of a suitable step-size requires (possibly) many projections onto C , see,e.g., [20, 29, 46]. We present our main result in this section. To this end, we begin with a result thatshows that the sequence { x n } generated by Algorithm 1 is bounded under the givenassumptions. Lemma 4.1.
Let { x n } be generated by Algorithm 1. Then under Assumptions 3.1and 3.2, we have that { x n } is bounded.Proof. Let x ∗ ∈ SOL. By Lemma 2.5 we get (since x ∗ ∈ C n ) that k x n +1 − x ∗ k = k P C n ( w n ) − x ∗ k ≤ k w n − x ∗ k − k x n +1 − w n k (12)= k w n − x ∗ k − dist ( w n , C n ) . Now, using Lemma 2.1 (c), we have k w n − x ∗ k = k (1 + α n )( x n − x ∗ ) − α n ( x n − − x ∗ ) k = (1 + α n ) k x n − x ∗ k − α n k x n − − x ∗ k + α n (1 + α n ) k x n − x n − k . (13)Substituting (13) into (12), we have k x n +1 − x ∗ k ≤ (1 + α n ) k x n − x ∗ k − α n k x n − − x ∗ k + α n (1 + α n ) k x n − x n − k −k x n +1 − w n k . (14)We also have (using Lemma 2.1 (a)) k x n +1 − w n k = k ( x x n +1 − x n ) − α n ( x n − x n − ) k = k x x n +1 − x n k + α k x n − x n − k − α n h x x n +1 − x n , x n − x n − i≥ k x x n +1 − x n k + α k x n − x n − k + α n (cid:16) − ρ n k x x n +1 − x n k − ρ n k x n − x n − k (cid:17) , (15)7here ρ n := α n + δσ . Combining (14) and (15), we get k x n +1 − x ∗ k − (1 + α n ) k x n − x ∗ k + α n k x n − − x ∗ k ≤ ( α n ρ n − k x n +1 − x n k + λ n k x n − x n − k , (16)where λ n := α n (1 + α n ) + α n − α n ρ n ρ n ≥ α n ρ n <
1. Taking into account the choice of ρ n , we have δ = 1 − α n ρ n σρ n and from (17), it follows that λ n = α n (1 + α n ) + α n − α n ρ n ρ n ≤ α (1 + α ) + ασδ. (18)Following the same arguments as in [1, 2, 11], we define ϕ n := k x n − x ∗ k , n ≥ ε n := ϕ n − α n ϕ n − + λ n k x n − x n − k , n ≥ . By the monotonicity of { α n } andthe fact that ϕ n ≥
0, we have ε n +1 − ε n ≤ ϕ n +1 − (1 + α n ) ϕ n + α n ϕ n − + λ n +1 k x n +1 − x n k − λ n k x n − x n − k . Using (16), we have ε n +1 − ε n ≤ ( α n ρ n − k x n +1 − x n k + λ n k x n − x n − k + λ n +1 k x n +1 − x n k − λ n k x n − x n − k = ( α n ρ n − λ n +1 ) k x n +1 − x n k . (19)We now claim that α n ρ n − λ n +1 ≤ − β. (20)Indeed by the choice of ρ n , we have α n ρ n − λ n +1 ≤ − β ⇔ α n ρ n − λ n +1 + β ≤ ⇔ λ n +1 + β + α n α n + δσ − ≤ ⇔ λ n +1 + β − δσα n + δσ ≤ ⇔ ( α n + δσ )( λ n +1 + β ) ≤ δσ α n + δσ )( λ n +1 + β ) ≤ (( α + δσ )( α (1 + α ) αδσ + β ) ≤ δσ, where the last inequality follows from Assumption 3.2 (b). Hence, the claim in (20)is true.Thus, it follows from (19) and (20) that ε n +1 − ε n ≤ − β k x n +1 − x n k . (21)The sequence { ε n } is non-increasing and the bounds of { α n } delivers − αϕ n − ≤ ϕ n − αϕ n − ≤ ε n ≤ ε , n ≥ . (22)It then follows that ϕ n ≤ α n ϕ + ε n − X k =0 α k ≤ α n ϕ + ε − α , n ≥ . (23)Combining (21) and (27), we get β n X k =1 k x k +1 − x k k ≤ ε − ε n +1 ≤ ε + αϕ n ≤ α n +1 ϕ + ε − α ≤ ϕ + ε − α , (24)which shows that ∞ X k =1 k x k +1 − x k k < ∞ . (25)Thus, lim n →∞ k x n +1 − x n k = 0. From w n = x n + α n ( x n − x n − ), we have k w n − x n k ≤ α n k x n − x n − k≤ α k x n − x n − k → , n → ∞ . Similarly, k x n +1 − w n k ≤ k x n +1 − x n k + k x n − w n k → , n → ∞ . Using Lemma 2.2, (16), (18) and (25), we have that lim n →∞ k x n − x ∗ k exists. Hence, { x n } is bounded.In the next two lemmas, we show that certain subsequences obtained in Algorithm 1are null subsequences. These two lemmas are necessary in order to show that theweak limit of { x n } is an element of SOL and for our weak convergence in Theorem 4.4below. 9 emma 4.2.
Let { x n } generated by Algorithm 1 above and Assumptions 3.1 and3.2 hold. Then (a) lim n →∞ η n k w n − z n k = 0 ; (b) lim n →∞ k w n − z n k = 0 . Proof.
Let x ∗ ∈ SOL. Since F is uniformly continuous on bounded subsets of X , then { F ( x n ) } , { z n } , { w n } and { F ( y n ) } are bounded. In particular, there exists M > k F ( y n ) k ≤ M for all n ∈ N . Combining Lemma 2.4 and Lemma 3.4,we get k x n +1 − x ∗ k = k P C n ( w n ) − x ∗ k ≤ k w n − x ∗ k − k x n +1 − w n k = k w n − x ∗ k − dist ( w n , C n ) ≤ k w n − x ∗ k − (cid:16) M h n ( w n ) (cid:17) ≤ k w n − x ∗ k − (cid:16) M ση n k r ( w n ) k (cid:17) = k w n − x ∗ k − (cid:16) M ση n k w n − z n k (cid:17) . (26)Since { x n } is bounded, we obtain from (26) that (cid:16) M ση n k w n − z n k (cid:17) ≤ k w n − x ∗ k − k x n +1 − x ∗ k = (cid:16) k w n − x ∗ k − k x n +1 − x ∗ k (cid:17)(cid:16) k w n − x ∗ k + k x n +1 − x ∗ k (cid:17) ≤ k w n − x ∗ k − k x n +1 − x ∗ k M ≤ k w n − x n +1 k M , (27)where M := sup n ≥ {k w n − x ∗ k + k x n +1 − x ∗ k} . This establishes (a).To establish (b), We distinguish two cases depending on the behaviour of (thebounded) sequence of step-sizes { η n } . Case 1 : Suppose that lim inf n →∞ η n >
0. Then0 ≤ k r ( w n ) k = η n k r ( w n ) k η n and this implies thatlim sup n →∞ k r ( w n ) k ≤ lim sup n →∞ (cid:18) η n k r ( w n ) k (cid:19)(cid:18) lim sup n →∞ η n (cid:19) = (cid:18) lim sup n →∞ η n k r ( w n ) k (cid:19) n →∞ η n = 0 . Hence, lim sup n →∞ k r ( w n ) k = 0. Therefore,lim n →∞ k w n − z n k = lim n →∞ k r ( w n ) k = 0 . ase 2 : Suppose that lim inf n →∞ η n = 0. Subsequencing if necessary, we mayassume without loss of generality that lim n →∞ η n = 0 and lim n →∞ k w n − z n k = a ≥ y n := γ η n z n + (cid:16) − γ η n (cid:17) w n or, equivalently, ¯ y n − w n = γ η n ( z n − w n ). Since { z n − w n } is bounded and since lim n →∞ η n = 0 holds, it follows thatlim n →∞ k ¯ y n − w n k = 0 . (28)From the step-size rule and the definition of ¯ y k , we have h F (¯ y n ) , w n − z n i < σ k w n − z n k , ∀ n ∈ N , or equivalently2 h F ( w n ) , w n − z n i + 2 h F (¯ y n ) − F ( w n ) , w n − z n i < σ k w n − z n k , ∀ n ∈ N . Setting t n := w n − F ( w n ), we obtain form the last inequality that2 h w n − t n , w n − z n i + 2 h F (¯ y n ) − F ( w n ) , w n − z n i < σ k w n − z n k , ∀ n ∈ N . Using Lemma 2.1 (b) we get2 h w n − t n , w n − z n i = k w n − z n k + k w n − t n k − k z n − t n k . Therefore, k w n − t n k − k z n − t n k < ( σ − k w n − z n k − h F (¯ y n ) − F ( w n ) , w n − z n i ∀ n ∈ N . Since F is uniformly continuous on bounded subsets of H and (28), if a > σ − a < n → ∞ . Fromthe last inequality we havelim sup n →∞ (cid:0) k w n − t n k − k z n − t n k (cid:1) ≤ ( σ − a < . For ǫ = − ( σ − a/ >
0, there exists N ∈ N such that k w n − t n k − k z n − t n k ≤ ( σ − a + ǫ = ( σ − a/ < ∀ n ∈ N , n ≥ N, leading to k w n − t n k < k z n − t n k ∀ n ∈ N , n ≥ N, which is a contradiction to the definition of z n = P C ( w n − F ( w n )). Hence a = 0,which completes the proof.The boundedness of the sequence { x n } implies that there is at least one weak limitpoint. We show that such weak limit point belongs to SOL in the next result.
Lemma 4.3.
Let Assumptions 3.1 and 3.2 hold. Furthermore let { x n k } be a subse-quence of { x n } converging weakly to a limit point p . Then p ∈ SOL. roof. By the definition of z n k together with (6), we have h w n k − F ( w n k ) − z n k , x − z n k i ≤ , ∀ x ∈ C, which implies that h w n k − z n k , x − z n k i ≤ h F ( w n k ) , x − z n k i , ∀ x ∈ C. Hence, h w n k − z n k , x − z n k i + h F ( w n k ) , z n k − w n k i ≤ h F ( w n k ) , x − w n k i , ∀ x ∈ C. (29)Fix x ∈ C and let k → ∞ in (39). Since lim k →∞ k w n k − z n k k = 0, we have0 ≤ lim inf k →∞ h F ( w n k ) , x − w n k i (30)for all x ∈ C . It follows from (39) and the monotonicity of F that h w n k − z n k , x − z n k i + h F ( w n k ) , z n k − w n k i ≤ h F ( w n k ) , x − w n k i≤ h F ( x ) , x − w n k i ∀ x ∈ C. Letting k → + ∞ in the last inequality, remembering that lim k →∞ k w n k − z n k k = 0for all k , we have h F ( x ) , x − p i ≥ ∀ x ∈ C. In view of Lemma 2.7, this implies p ∈ SOL.All is now set to give the weak convergence result in the theorem below.
Theorem 4.4.
Let Assumptions 3.1 and 3.2 hold. Then the sequence { x n } generatedby Algorithm 1 weakly converges to a point in SOL.Proof. We have shown that(i) lim n →∞ k x n − x ∗ k exists;(ii) ω w ( x n ) ⊂ SOL, where ω w ( x n ) := { x : ∃ x n j ⇀ x } denotes the weak ω -limit setof { x n } .Then, by Lemma 2.3, we have that { x n } converges weakly to a point in SOL.We give some discussions on further contributions of this paper in the remark below. Remark 4.5. (a) Our iterative Algorithm 1 is more applicable than some recent re-sults on projection type methods with inertial extrapolation step for solving VI(
F, C )(1) in real Hilbert spaces. For instance, the proposed method in [21] can only beapplied for a case when F is monotone and L -Lipschitz continuous. Moreover, theLipschitz constant or an estimate of it has to be known when implementing theAlgorithm 3.1 of [21]. In this result, Algorithm 1 is applicable when F is uniformlycontinuous and monotone operator.(b) In finite-dimensional spaces, the assumption that F is uniformly continuouson bounded subsets of C automatically holds when F is continuous. Moreover, inthis case, only continuity of F is required and our weak convergence in Theorem 4.4coincides with global convergence of sequence of iterates { x n } in R n .(c) Lemmas 3.5, 4.1, 4.2 and Theorem 4.4 still hold for a more general case of F pseudo-monotone (i.e., for all x, y ∈ H , h F ( x ) , y − x i ≥ ⇒ h F ( y ) , y − x i ≥ F pseudo-monotone in the Appendix. ♦ Rate of Convergence
In this section we give the rate of convergence of the iterative method 1 proposed inSection 3. We show that the proposed method has sublinear rate of convergence andestablish the nonasymptotic O (1 /n ) convergence rate of the proposed method. Tothe best of our knowledge, there is no convergence rate result known in the literaturewithout stronger assumptions for inertial projection-type Algorithm 1 for VI( F, C )(1) in infinite dimensional Hilbert spaces.
Theorem 5.1.
Let Assumptions 3.1 and 3.2 hold. Let the sequence { x n } be gen-erated by Algorithm 1 and x = x . Then for any x ∗ ∈ SOL and for any positiveinteger n , it holds that min ≤ i ≤ n k x i +1 − w i k ≤ h (cid:16) α − α + α (1 − α ) β (cid:17)(cid:16) − α − α (cid:17)i k x − x ∗ k n . Proof.
From (14), we have k x n +1 − x ∗ k − k x n − x ∗ k − α n ( k x n − x ∗ k − k x n − − x ∗ k ) ≤ α n (1 + α n ) k x n − x n − k − k x n +1 − w n k . (31)This implies that k x n +1 − w n k ≤ ϕ n − ϕ n +1 + α n ( ϕ n − ϕ n − ) + δ n ≤ ϕ n − ϕ n +1 + α [ V n ] + + δ n , (32)where δ n := α n (1 + α n ) k x n − x n − k , V n := ϕ n − ϕ n − , [ V n ] + := max { V n , } and ϕ n := k x n − x ∗ k .Observe from (24) that ∞ X n =1 k x n +1 − x n k ≤ β h ϕ + ε − α i . So, ∞ X n =1 δ n = ∞ X n =1 α n (1 + α n ) k x n − x n − k ≤ ∞ X n =1 α (1 + α ) k x n − x n − k = α (1 + α ) ∞ X n =1 k x n − x n − k ≤ α (1 + α ) β h ϕ + ε − α i := C . (33)The inequality (31) implies that V n +1 ≤ α n V n + δ n α [ V n ] + + δ n . Therefore, [ V n +1 ] + ≤ α [ V n ] + + δ n ≤ α n [ V ] + + n X j =1 α j − δ n +1 − j . (34)Note that by our assumption x = x . This implies that V = [ V ] + = 0 and δ = 0.From (34), we get ∞ X n =2 [ V n ] + ≤ − α ∞ X n =1 δ n = 11 − α ∞ X n =2 δ n . (35)From (32), we get n X i =1 k x i +1 − w i k ≤ ϕ − ϕ n + α n X i =1 [ V i ] + + n X i =2 δ i ≤ ϕ + αC + C , (36)where C = C − α ≥ − α P ∞ i =2 δ i ≥ P ∞ i =2 [ V i ] + by (35). Now, since ε = ϕ − α ϕ =(1 − α ) ϕ , we have ϕ + αC + C = ϕ + αC − α + α (1 + α ) β h ϕ + ε − α i = ϕ + αC − α + α (1 + α ) β h − α − α i ϕ = ϕ + α − α h − α − α i + α (1 + α ) β h − α − α i ϕ = h (cid:16) α − α + α (1 − α ) β (cid:17)(cid:16) − α − α (cid:17)i ϕ . (37)From (36) and (37), we obtainmin ≤ i ≤ n k x i +1 − w i k ≤ h (cid:16) α − α + α (1 − α ) β (cid:17)(cid:16) − α − α (cid:17)i k x − x ∗ k n . (38)14 emark 5.2. (a) Note that x n +1 = w n implies that w n ∈ C n , where C n is as definedin Algorithm 1 and hence h n ( w n ) ≤
0. By Lemma 3.4, we get ση n k w n − z n k ≤ h n ( w n ). Therefore, 0 ≤ ση n k w n − z n k ≤ h n ( w n ) ≤ , which implies that w n = z n . Thus, the equality x n +1 = w n implies that x n +1 isalready a solution of VI( F, C ) (1). In this sense, the error estimate given in Theorem5.1 can be viewed as a convergence rate result of the inertial projection-type method1. In particular, (38) implies that, to obtain an ǫ -optimal solution in the sense that k x n +1 − w n k < ǫ , the upper bound of iterations required by inertial projection-typemethod 1 is h (cid:16) α − α + α (1 − α ) β (cid:17)(cid:16) − α − α (cid:17)i k x − x ∗ k ǫ . We note that with the ” min ≤ i ≤ n ”,a nonasymptotic O (1 /n ) convergence rate implies that an ǫ -accuracy solution, in thesense that k x n +1 − w n k < ǫ , is obtainable within no more than O (1 /ǫ ) iterations.Furthermore, if α n = 0 for all n , then the ”min ≤ i ≤ n ” can be removed by setting i = n in Theorem 5.1. ♦ In this section, we discuss the numerical behaviour of Algorithm 1 using differenttest examples taken from the literature which are describe below and compare ourmethod with (2), (3) and the original Algorithm (when α n = 0) of Algorithm 1. Example 6.1.
This first example (also considered in [35, 36]) is a classical examplefor which the usual gradient method does not converge. It is related to the uncon-strained case of VI(
F, C ) (1) where the feasible set is C := R m (for some positiveeven integer m ) and F := ( a ij ) ≤ i,j ≤ m is the square matrix m × m whose terms aregiven by a ij = − , if j = m + 1 − i and j > i , if j = m + 1 − i and j < i z = (0 , . . . ,
0) is the solution of this test example.The initial point x is the unit vector. We choose γ = 0 . σ = 0 . α n = 0 . m = 500.The numerical result is listed in Figure 1, which illustrates that Algorithm 1 highlyimproves the original Algorithm. Example 6.2.
This example is taken from [24] and has been considered by manyauthors for numerical experiments (see, for example, [26, 37, 43]). The operator A is defined by A ( x ) := M x + q , where M = BB T + S + D , where B, S, D ∈ R m × m are randomly generated matrices such that S is skew-symmetric (hence the operatordoes not arise from an optimization problem), D is a positive definite diagonal matrix(hence the variational inequality has a unique solution) and q = 0. The feasible set15
50 100 150 200 250 300
Iteration -40 -30 -20 -10 k x n k Algorithm 1The original Algorithm
Figure 1: Comparison of Algorithm 1 with the original Algorithm. C is described by linear inequality constraints Bx ≤ b for some random matrix B ∈ R k × m and a random vector b ∈ R k with nonnegative entries. Hence the zerovector is feasible and therefore the unique solution of the corresponding variationalinequality. These projections are computed by solving a quadratic optimizationproblem using the MATLAB solver quadprog . Hence, for this class of problems,the evaluation of A is relatively inexpensive, whereas projections are costly. Wepresent the corresponding numerical results (number of iterations and CPU times inseconds) using four different dimensions m and two different numbers of inequalityconstraints k .We compare our proposed Algorithm 1, original Algorithm, subgradient extragradi-ent method (2) and the inertial subgradient extragradient method (3) using Example6.2 and the numerical results are listed in Tables 1 -4 and shown in Figures 2-5 be-low. We take the initial point x to be the unit vector in these algorithms. Weuse “OPM” to denote the original Algorithm, “SPM” to denote the subgradientextragradient method (2) and “iSPM” to denote inertial subgradient extragradientmethod (3).We choose the stopping criterion as k x k k ≤ ǫ = 0 . . The size k = 30 , ,
80 and m = 20 , , , B, S, D and the vector b are generated randomly.We choose γ = 0 . σ = 0 . α n = 0 . σ = 0 . ρ = 0 . µ = 0 .
2. In iSPM (3), α = 0 . , L = k M k , τ = 0 . − α − α − α + α .We denote by “Iter.” the number of iterations and “InIt.” the number of totaliterations of finding suitable step size in Tables 1 -4 below.16able 1: Comparison of Algorithm 1, original Algorithm and methods (2) and (3)for k = 50 , α n = 0 . . Iter. InIt. CPU in second m Alg.1 OPM SPM iSPM Alg.1 OPM SPM Alg.1 OPM SPM iSPM20 1044 1162 2891 4783 2376 5022 4283 0.0781 0.2188 0.6563 0.578150 4829 5912 25544 29639 13809 30885 123982 0.5625 0.6094 7.5938 2.187580 19803 22129 19736 61322 74066 157061 93047 6.6094 7.0313 15.9063 13.4844100 26821 31149 35520 92579 101925 220297 173670 12.1406 15.8438 67.4219 34.6250
Table 2: Comparison of Algorithm 1, the original Algorithm and methods (2) and(3) for k = 80 , α n = 0 . . Iter. InIt. CPU in second m Alg.1 OPM SPM iSPM Alg.1 OPM SPM Alg.1 OPM SPM iSPM20 1479 1651 5096 5306 3676 7783 19071 0.1875 0.2188 6.7869 0.750050 4152 5088 8144 28033 11955 26594 36342 1.0625 1.2344 8.2344 6.281380 22711 25864 22281 64588 85176 182177 105237 8.2188 9.3438 20.6719 16.1563100 26314 30568 37588 88138 99998 216162 185430 13.0625 16.1250 118.4375 36.0625
Table 3: Comparison of Algorithm 1, the original Algorithm and methods (2) and(3) for k = 30 , α n = 0 . . Iter. InIt. CPU in second m Alg.1 OPM SPM iSPM Alg.1 OPM SPM Alg.1 OPM SPM iSPM10 197 469 509 1327 341 1370 1460 1.21884 0.0313 0.2969 0.250030 1548 4745 4347 13697 4243 17534 16880 10.8750 0.4063 1.0938 0.875050 1581 4899 8269 26393 4762 18859 37237 12.9688 0.5000 2.4688 1.609470 6192 6256 7826 56491 22381 81445 31406 62.4844 5.5781 9.3281 9.1813
Table 4: Comparison of Algorithm 1, the original Algorithm and methods (2) and(3) for k = 50 , α n = 0 . . Iter. InIt. CPU in second m Alg.1 OPM SPM iSPM Alg.1 OPM SPM Alg.1 OPM SPM iSPM10 147 419 579 1117 253 1024 1119 3 0.0313 0.5781 0.359430 1715 5110 6373 13934 4705 19018 25006 30.0469 0.4844 1.5469 1.156350 1308 4798 9227 31585 4062 17873 41084 41.4375 0.4844 3.5156 2.046970 5673 14944 8205 52124 20548 74974 33716 393 4.5469 7.6406 10.4375
Tables 1 -4 and Figures 2-5 show that Algorithm 1 improves the original Algorithmwith respect to “Iter.”, “InIt.” and CPU time. It is also observed from Tables1 -4 and Figures 2-5 that our proposed Algorithm 1 outperform the subgradientextragradient method (2) and the inertial subgradient extragradient method (3)with respect to the CPU time and the number of iterations when the feasible set C is nonempty closed affine subset of H . 17
100 200 300 400 500 600 700 800 900 1000
Iteration -2 -1 k x n k Alg 1OPMiSPMSPM
Figure 2: Comparison of Algorithm 1, original Algorithm and methods (2) and (3). k = 50 , α n = 0 . . Iteration -2 -1 k x n k Alg 1OPMiSPMSPM
Figure 3: Comparison of Algorithm 1, original Algorithm and methods (2) and (3). k = 80 , α n = 0 . . This paper presents a weak convergence result with inertial projection-type methodfor monotone variational inequality problems in real Hilbert spaces under very mildassumptions. This class of method is of inertial nature because at each iteration the18
100 200 300 400 500 600 700 800 900 1000
Iteration -3 -2 -1 k x n k Alg 1OPMiSPMSPM
Figure 4: Comparison of Algorithm 1, original Algorithm and methods (2) and (3). k = 30 , α n = 0 . . Iteration -3 -2 -1 k x n k Alg 1OPMiSPMSPM
Figure 5: Comparison of Algorithm 1, original Algorithm and methods (2) and (3). k = 50 , α n = 0 . . projection-type is applied to a point extrapolated at the current iterate in the direc-tion of last movement. Our proposed algorithm framework is not only more simpleand intuitive, but also more general than some already proposed inertial projectiontype methods for solving variational inequality. Based on some pioneering analysisand Algorithm 1, we established certain nonasymptotic O (1 /n ) convergence rate19esults. Our preliminary implementation of the algorithms and experimental resultshave shown that inertial algorithms are generally faster than the corresponding orig-inal un-accelerated ones. In our experiments, the extrapolation step-length α n wasset to be constant. How to select α n adaptively such that the overall performance isstable and more efficient deserves further investigation. Interesting topics for futureresearch may include relaxing the conditions on { α n } , improving the convergenceresults, and proposing modified inertial-type algorithms so that the extrapolationstep-size can be significantly enlarged. Acknowledgements
The project of the first author has received funding from theEuropean Research Council (ERC) under the European Union’s Seventh FrameworkProgram (FP7 - 2007-2013) (Grant agreement No. 616160)
References [1] F. Alvarez, Weak convergence of a relaxed and inertial hybrid projection-proximal point algorithm for maximal monotone operators in Hilbert space.
SIAM J. Optim. (2004), 773-782.[2] F. Alvarez and H. Attouch; An inertial proximal method for maximal monotoneoperators via discretization of a nonlinear oscillator with damping, Set-ValuedAnal. (2001), 3-11.[3] H. Attouch, X. Goudon and P. Redont; The heavy ball with friction. I. Thecontinuous dynamical system, Commun. Contemp. Math. (1) (2000), 1-34.[4] H. Attouch and M.O. Czarnecki; Asymptotic control and stabilization of non-linear oscillators with non-isolated equilibria, J. Differential Equations (1)(2002), 278-310.[5] H. Attouch, J. Peypouquet and P. Redont; A dynamical approach to an iner-tial forward-backward algorithm for convex minimization,
SIAM J. Optim. (2014), 232-256.[6] H. Attouch and J. Peypouquet; The rate of convergence of Nesterov’s acceler-ated forward-backward method is actually faster than k , SIAM J. Optim. (2016), 1824-1834.[7] J.-P. Aubin and I. Ekeland; Applied Nonlinear Analysis , Wiley, New York, 1984.[8] C. Baiocchi and A. Capelo;
Variational and Quasivariational Inequalities; Ap-plications to Free Boundary Problems , Wiley, New York (1984).[9] H.H. Bauschke and P.L. Combettes;
Convex Analysis and Monotone OperatorTheory in Hilbert Spaces , CMS Books in Mathematics, Springer, New York(2011).[10] A. Beck and M. Teboulle; A fast iterative shrinkage-thresholding algorithm forlinear inverse problems,
SIAM J. Imaging Sci. (1) (2009), 183-202.2011] R. I. Bot, E. R. Csetnek and C. Hendrich; Inertial Douglas-Rachford splittingfor monotone inclusion, Appl. Math. Comput. (2015), 472-487.[12] R. I. Bot and E. R. Csetnek; An inertial alternating direction method of mul-tipliers,
Minimax Theory Appl. (2016), 29-49.[13] R. I. Bot and E. R. Csetnek; An inertial forward-backward-forward primal-dualsplitting algorithm for solving monotone inclusion problems, Numer. Alg. (2016), 519-540.[14] A. Cegielski; Iterative Methods for Fixed Point Problems in Hilbert Spaces ,Lecture Notes in Mathematics 2057, Springer, Berlin, 2012.[15] L.C. Ceng, N. Hadjisavvas, and N.-C. Wong; Strong convergence theorem bya hybrid extragradient-like approximation method for variational inequalitiesand fixed point problems,
J. Glob. Optim. (2010), 635-646.[16] L. C. Ceng and J. C. Yao; An extragradient-like approximation method forvariational inequality problems and fixed point problems, Appl. Math. Comput. (2007), 205-215.[17] Y. Censor, A. Gibali, and S. Reich; Strong convergence of subgradient extragra-dient methods for the variational inequality problem in Hilbert space,
Optim.Methods Softw. (2011), 827-845.[18] Y. Censor, A. Gibali and S. Reich; The subgradient extragradient method forsolving variational inequalities in Hilbert space, J. Optim. Theory Appl. , (2011), 318-335.[19] C. Chen, R. H. Chan, S. Ma and J. Yang; Inertial Proximal ADMM forLinearly Constrained Separable Convex Optimization, SIAM J. Imaging Sci. (2015), 2239-2267.[20] S. Denisov, V. Semenov and L. Chabak; Convergence of the modified extragra-dient method for variational inequalities with non-Lipschitz operators, Cyber-net. Systems Anal. (2015), 757-765.[21] Q. L. Dong, Y. J. Cho, L. L. Zhong and Th. M. Rassias; Inertial projection andcontraction algorithms for variational inequalities, J. Global Optim. (2018),687-704.[22] R. Glowinski, J.-L. Lions, and R. Tr´emoli`eres; Numerical Analysis of Varia-tional Inequalities , North-Holland, Amsterdam (1981).[23] K. Goebel and S. Reich;
Uniform convexity, hyperbolic geometry, and nonex-pansive mappings , Marcel Dekker, New York, (1984).[24] P. T. Harker, and J.-S. Pang; A damped-Newton method for the linear comple-mentarity problem, in Computational Solution of Nonlinear Systems of Equa-tions, Lectures in Appl. Math. 26, G. Allgower and K. Georg, eds., AMS,Providence, RI, 1990, pp. 265-284. 2125] Y. R. He; A new double projection algorithm for variational inequalities,
J.Comput. Appl. Math. (2006), 166-173.[26] D. V. Hieu, P. K. Anh and L. D. Muu; Modified hybrid projection methods forfinding common solutions to variational inequality problems,
Comput. Optim.Appl. (2017), 75-96[27] A.N. Iusem and R. G´arciga Otero; Inexact versions of proximal point and aug-mented Lagrangian algorithms in Banach spaces, Numer. Funct. Anal. Optim. (2001), 609-640.[28] A.N. Iusem and M. Nasri; Korpelevich’s method for variational inequality prob-lems in Banach spaces, J. Global Optim. (2011), 59-76.[29] E.N. Khobotov; Modification of the extragradient method for solving varia-tional inequalities and certain optimization problems, USSR Comput. Math.Math. Phys. (1989), 120-127.[30] D. Kinderlehrer and G. Stampacchia; An Introduction to Variational Inequali-ties and Their Applications , Academic Press, New York (1980).[31] I.V. Konnov;
Combined Relaxation Methods for Variational Inequalities ,Springer-Verlag, Berlin (2001).[32] G.M. Korpelevich; The extragradient method for finding saddle points andother problems, ´Ekon. Mat. Metody (1976), 747-756.[33] D. A. Lorenz and T. Pock; An inertial forward-backward algorithm for mono-tone inclusions, J. Math. Imaging Vis. , (2015), 311-325.[34] P. E. Maing´ e ; Regularized and inertial algorithms for common fixed points ofnonlinear operators, J. Math. Anal. Appl. (2008), 876-887.[35] P.-E. Maing´e and M.L. Gobinddass; Convergence of one-step projected gradientmethods for variational inequalities,
J. Optim. Theory Appl. (2016), 146-168.[36] Yu.V. Malitsky; Projected reflected gradient methods for monotone variationalinequalities,
SIAM J. Optim. (2015), 502-520.[37] Yu.V. Malitsky and V.V. Semenov; A hybrid method without extrapolationstep for solving variational inequality problems, J. Global Optim. (2015),193-202.[38] P. Marcotte; Applications of Khobotov’s algorithm to variational and networkequlibrium problems, Inf. Syst. Oper. Res. (1991), 258-270.[39] J. Mashreghi and M. Nasri; Forcing strong convergence of Korpelevich’s methodin Banach spaces with its applications in game theory, Nonlinear Analysis (2010), 2086-2099. 2240] N. Nadezhkina and W. Takahashi; Strong convergence theorem by a hybridmethod for nonexpansive mappings and Lipschitz-continuous monotone map-pings, SIAM J. Optim. (2006), 1230-1241.[41] P. Ochs, T. Brox and T. Pock; iPiasco: Inertial Proximal Algorithm forstrongly convex Optimization, J. Math. Imaging Vis. (2015), 171-181.[42] B. T. Polyak; Some methods of speeding up the convergence of iterarive meth-ods, Zh. Vychisl. Mat. Mat. Fiz. (1964), 1-17.[43] M.V. Solodov and B.F. Svaiter; A new projection method for variational in-equality problems, SIAM J. Control Optim. (1999), 765-776.[44] W. Takahashi; Nonlinear Functional Analysis , Yokohama Publishers, Yoko-hama, (2000).[45] D. V. Thong and D. V. Hieu; Modified subgradient extragradient method forvariational inequality problems.
In press: Numer. Algor. doi:10.1007/s11075-017-0452-4.[46] P. Tseng; A modified forward-backward splitting method for maximal monotonemappings,
SIAM J. Control Optim. (2000), 431-446. In this case, we present a version of Lemma 4.3 for the case when F is pseudo-monotone. Lemma 8.1.
Let F be pseudo-monotone, uniformly continuous and sequentiallyweakly continuous on H . Assume that Assumption 3.2 holds. Furthermore let { x n k } be a subsequence of { x n } converging weakly to a limit point p . Then p ∈ SOL.Proof.
By the definition of z n k together with (6), we have h w n k − F ( w n k ) − z n k , x − z n k i ≤ , ∀ x ∈ C, which implies that h w n k − z n k , x − z n k i ≤ h F ( w n k ) , x − z n k i , ∀ x ∈ C. Hence, h w n k − z n k , x − z n k i + h F ( w n k ) , z n k − w n k i ≤ h F ( w n k ) , x − w n k i , ∀ x ∈ C. (39)Fix x ∈ C and let k → ∞ in (39). Since lim k →∞ k w n k − z n k k = 0, we have0 ≤ lim inf k →∞ h F ( w n k ) , x − w n k i (40)for all x ∈ C . Now we choose a sequence { ǫ k } k of positive numbers decreasing andtending to 0. For each ǫ k , we denote by N k the smallest positive integer such that (cid:10) F ( w n j ) , x − w n j (cid:11) + ǫ k ≥ ∀ j ≥ N k , (41)23here the existence of N k follows from (40). Since { ǫ k } is decreasing, it is easy tosee that the sequence { N k } is increasing. Furthermore, for each k , F ( w N k ) = 0 and,setting v N k = F ( w N k ) k F ( w N k k , we have h F ( w N k ) , v N k i = 1 for each k . Now we can deduce from (41) that for each k h F ( w N k ) , x + ǫ k v N k − w N k i ≥ , and, since F is pseudo-monotone, that h F ( x + ǫ k v N k ) , x + ǫ k v N k − w N k i ≥ . (42)On the other hand, we have that { x n k } converges weakly to p when k → ∞ . Since F is sequentially weakly continuous on C , { F ( w n k ) } converges weakly to F ( p ). Wecan suppose that F ( p ) = 0 (otherwise, p is a solution). Since the norm mapping issequentially weakly lower semicontinuous, we have0 < k F ( p ) k ≤ lim inf k →∞ k F ( w n k ) k . Since { w N k } ⊂ { w n k } and ǫ k → k → ∞ , we obtain0 ≤ lim sup k →∞ k ǫ k v N k k = lim sup k →∞ (cid:16) ǫ k k F ( w n k ) k (cid:17) ≤ lim sup k →∞ ǫ k lim inf k →∞ k F ( w n k ) k ≤ k F ( p ) k = 0 , which implies that lim k →∞ k ǫ k v N k k = 0. Hence, taking the limit as k → ∞ in (42),we obtain h F ( x ) , x − p i ≥ . Now, using Lemma 2.2 of [39], we have that p ∈∈