Minimizing curves in prox-regular subsets of Riemannian manifolds
aa r X i v : . [ m a t h . DG ] F e b MINIMIZING CURVES IN PROX-REGULAR SUBSETS OFRIEMANNIAN MANIFOLDS
MOHAMAD R. POURYAYEVALI AND HAJAR RADMANESH
Abstract.
We obtain a characterization of the proximal normal coneto a prox-regular subset of a Riemannian manifold. Moreover, someproperties of Bouligand tangent cones to prox-regular sets are described.We prove that for a prox-regular subset S of a Riemannian manifold, themetric projection P S to S is locally Lipschitz on an open neighborhood of S and it is directionally differentiable at boundary points of S . Finally,a necessary condition for a curve to be a minimizing curve in a prox-regular set is derived. Introduction
Closed subsets of Hilbert spaces satisfying an external sphere conditionwith uniform radius have been studied as generalizations of convex sets,mostly in relation to uniqueness of the metric projection and smoothnessof the distance function. In the fundamental paper [10] where the finitedimensional case is considered, these sets were called sets with positivereach. Then various equivalent definitions related to this property havebeen presented independently by several authors; see [7, 19] and the refer-ences therein. Among them, one can mention the notions of ϕ -convexity (astitled p -convexity) and prox-regularity of sets which were introduced in [8]and [19], respectively. It was shown in [5] that certain properties which holdglobally for convex sets are still valid locally for ϕ -convex sets.Differentiability properties of the metric projection onto closed convex setsare of interest in sensitivity analysis of variational inequalities and optimalcontrol problems. Moreover, the regularity of the metric projection ontoa sufficiently regular submanifold M of R n as well as the regularity of thecorresponding distance function have significant role in various aspects ofanalysis; see [18]. A classical example is the Dirichlet problem for quasilinearpartial differential equations, where the manifold of interest is the boundaryof the underlying domain; see, for instance, [11].The example presented by J. Kruskal [15] shows that, in general sucha projection is not directionally differentiable, even in finite dimensional Mathematics Subject Classification.
Key words and phrases. prox-regular sets, ϕ -convex sets, metric projection, non-smooth analysis, Riemannian manifolds. spaces. By directionally differentiable at a point we mean that the direc-tional derivative exists for all directions through that point. This is weakerthan the existence of the gradient at that point.The problem of differentiability of the metric projection for a closed locallyconvex subset S of a finite dimensional Riemannian manifold M was studiedin [23] and it was proved in [12] that for a closed totally convex subset S ⊂ M , there exists an open set W containing S such that the metricprojection is locally Lipschitz on W .In [3] the notion of ϕ -convex sets was extended to Hadamard manifoldsand it was shown that if S is a ϕ -convex subset of an infinite-dimensionalHadamard manifold M , then there exists a neighborhood U of S in M suchthat the metric projection P S : U → S is single-valued and locally Lipschitz.Moreover, it was proved that under the same assumptions on S and M , thereexists a neighborhood U of S in M such that d S is C with locally Lipschitzgradient on U \ S . On the other hand, in [13] the notion of prox-regular setswas introduced on Riemannian manifolds as a subclass of regular sets. In[20] we proved that the two classes of ϕ -convex sets and prox-regular setscoincide in the setting of Riemannian manifolds.The problem of existence and uniqueness of geodesics on a Riemannianmanifold without boundary is a classical subject of differential geometry andglobal nonlinear analysis and is particularly fit to a treatment by variationalmethods. However, in the case of Riemannian manifolds with boundary orcertain subsets of a manifold without boundary, strong irregularities appearin the energy functional and new techniques are needed for dealing with theseproblems. In [4, 5] ϕ -convex subsets of a Real Hilbert space were consideredand using an infinitesimal definition of geodesics in the framework of Sobolevspaces the author characterized these geodesics as critical points of an energyfunctional on a suitable path spaces.The aim of this paper is to study minimizing curves in a prox-regular sub-set S of a Riemannian manifold M . To this end, we use some powerful toolsfrom nonsmooth analysis and an adapted variational technique. Applyingthe first variation formula, we give a necessary condition for an admissiblecurve γ : [ a, b ] → M in S to be minimizing. Indeed, this curve has theproperty that D t ˙ γ ( t ) ∈ N PS ( γ ( t )) , for every t ∈ [ a, b ] except for finitely many points, provided that S has a C boundary, where N PS ( x ) is the proximal normal cone at x ∈ S . To prove thisresult, we address the problem of the directional differentiability of the met-ric projection P S at boundary points of S . Employing Shapiro’s variationalprinciple [22], we show that for a prox-regular subset S of a Riemannianmanifold M , the projection map P S is locally Lipschitz on an open neigh-borhood of S which generalizes the result of [3] to the Riemannian setting.Moreover, we prove that P S is directionally differentiable at boundary points INIMIZING CURVES IN PROX-REGULAR SETS 3 of S . We also obtain a useful characterization of Bouligand tangent cone toa prox-regular set.The paper is organized as follows. In Section 2 we present some basicconstructions and preliminaries in Riemannian geometry and nonsmoothanalysis, widely used in the sequel. Section 3 is devoted to the study ofBouligand and proximal normal cones. Then we obtain a characterizationof the proximal normal cone to a prox-regular set. We also show that P S is alocally Lipschitz retraction from a neighborhood of S to S . In Section 4 dif-ferentiability properties of the metric projection P S to a prox-regular subset S of a Riemannian manifold are investigated which leads to a characteriza-tion of Bouligand tangent cone. Section 5 is concerned with the necessarycondition for a curve γ to be a minimizing curve in a prox-regular set whoseboundary is a C submanifold of M . Moreover, some relevant examples arepresented. 2. Preliminaries and notations
Let us recall some notions of Riemannian manifolds and nonsmooth analy-sis; see, e.g., [6, 9, 21]. Throughout this paper, (
M, g ) is a finite-dimensionalRiemannian manifold endowed with a Riemannian metric g x = h ., . i x oneach tangent space T x M and ∇ is the Riemannian connection of g . Forevery x, y ∈ M , the Riemannian distance from x to y is denoted by d ( x, y ).Moreover, B ( x, r ) and B ( x, r ) signify the open and closed metric ball cen-tered at x with radius r , respectively. For a smooth curve γ : I → M and t , t ∈ I , the notation L γt t is used for the parallel transport along γ from γ ( t ) to γ ( t ). When γ is the unique minimizing geodesic joining γ ( t ) to γ ( t ), we use L t t instead of L γt t . Furthermore for a smooth vector field X along γ , D t X is the covariant derivative of X along γ .For x ∈ M , let r ( x ) be the convexity radius at x . Then the function x r ( x ) from M to R + ∪{ + ∞} is continuous; see [21]. The map exp x : U x → M will stand for the exponential map at x , where U x is an open subset of thetangent space T x M containing 0 x ∈ T x M . Note that if x and y belong to aconvex set, then both exp − x y and exp − y x are defined and k exp − x y k = d ( x, y ) = k exp − y x k . Moreover, L xy (cid:0) exp − x y (cid:1) = − exp − y x. For a fixed point z ∈ M , the function φ : M → R defined by φ ( x ) = d ( x, z ) is C ∞ on any convex neighborhood of z and for every x in a convexneighborhood of z , ∇ φ ( x ) = − − x z .Let S be a nonempty closed subset of M . The proximal normal cone to S at x ∈ S , is denoted by N PS ( x ) and ξ ∈ N PS ( x ) if and only if there exists σ > h ξ, exp − x y i ≤ σ d ( x, y ) , M. R. POURYAYEVALI AND H. RADMANESH for every y ∈ U ∩ S , where U is a convex neighborhood of x . The metricprojection to S , denoted by P S , is defined by P S ( z ) = (cid:26) x ∈ S : d ( x, z ) = inf y ∈ S d ( y, z ) (cid:27) ∀ z ∈ M. Moreover, Unp( S ) is considered as the set of all points z ∈ M with theproperty that P S ( z ) is single-valued. Then according to [20, Lemma 4.11],the projection map P S : Unp( S ) → S is continuous. For every x ∈ S wealso define reach( S, x ) := sup { r ≥ B ( x, r ) ⊆ Unp( S ) } , It is worth mentioning that the function x reach( S, x ) is continuous on S ; see [2, 14] for more details.In order to deduce the Lipschitz property and directional differentiabilityof P S , we use the following variational principal by A. Shapiro [22]. Let f, g : X → R be two functions on a Hilbert space X and S, T ⊂ X . Considerthe optimization problems(2.1) min x ∈ S f ( x )and(2.2) min x ∈ T g ( x ) . Let x and ¯ x be some optimal solutions of (2.1) and (2.2), respectively andsuppose that there exist a neighborhood W of x and α > x ∈ S ∩ W ,(2.3) f ( x ) ≥ f ( x ) + α k x − x k . Also, suppose that ¯ x ∈ W and f and g are Lipschitz on W with Lipschitzconstants k and k , respectively. Then(2.4) k ¯ x − x k ≤ α − κ + 2 δ + α − / ( k δ + k δ ) / , where κ is a Lipschitz constant of h ( x ) = g ( x ) − f ( x ) on W and δ = sup x ∈ T ∩ W d ( x, S ∩ W ) ,δ = d ( x , T ∩ W ) . Local Lipschitzness of metric projection
In this section we first derive some properties of Bouligand tangent conesto prox-regular sets which we need in the sequel. Let us begin by recallingsome required definitions; see [13, 20].The closed subset S of M is said to be prox-regular at ¯ x ∈ S if there exist ε > σ > B (¯ x, ε ) is convex and for every x ∈ S ∩ B (¯ x, ε )and v ∈ N PS ( x ) with k v k < ǫ , h v, exp − x y i ≤ σ d ( x, y ) for every y ∈ S ∩ B (¯ x, ǫ ) . INIMIZING CURVES IN PROX-REGULAR SETS 5
Moreover, S is called prox-regular if it is prox-regular at each point of S .In [20, Theorem 3.4], we proved that every ϕ -convex subset of a Riemann-ian manifold M is prox-regular and conversely, for every prox-regular subset S of M there exists a continuous function ϕ : S → [0 , ∞ ) such that S is ϕ -convex. Recall that a closed subset S ⊂ M is called ϕ -convex if for every x ∈ S and v ∈ N PS ( x ) h v, exp − x y i ≤ ϕ ( x ) k v k d ( x, y ) , for every y ∈ U ∩ S , where U is a convex neighborhood of x and ϕ : S → [0 , ∞ ) is a continuous function. Note that this definition is independent ofthe choice of any convex neighborhood of x .Let S ⊂ M be a closed subset and x ∈ S . The Bouligand (or contingent)tangent cone to S at x is defined as T BS ( x ) := (cid:26) lim i →∞ exp − x z i t i : z i ∈ U ∩ S, z i → x and t i ↓ (cid:27) , where U is a convex neighborhood of x in M . It was shown in [13] thatwhen S is prox-regular, T BS ( x ) is a convex cone for every x ∈ S . Lemma 3.1.
Let S ⊆ M be a prox-regular set and x ∈ S . Then (i) T BS ( x ) = (cid:0) N PS ( x ) (cid:1) ◦ , (ii) (cid:0) T BS ( x ) (cid:1) ◦ = N PS ( x ) .Proof. Assertion (i) can be obtained from [13, Lemma 3.7]. Indeed, we have T CS ( x ) ⊆ T BS ( x ) ⊆ (cid:0) N PS ( x ) (cid:1) ◦ = (cid:0) N CS ( x ) (cid:1) ◦ = T CS ( x ) , where T CS ( x ) and N CS ( x ) are (Clarke) tangent and normal cone to S at x ,respectively.Assertion (ii) follows from the fact that N PS ( x ) is closed and convex.Hence (cid:0)(cid:0) N PS ( x ) (cid:1) ◦ (cid:1) ◦ = N PS ( x ). (cid:3) (cid:3) According to [20, Proposition 4.2], for every point x in a closed prox-regular subset S of M , reach( S, x ) >
0. This property of prox-regular setshelps us to prove the following topological property of these sets.
Lemma 3.2. If S is a closed set with the property that reach( S, x ) > forevery x ∈ S , then S is locally connected.Proof. Let x ∈ S and U be an open neighborhood of x in M . We are goingto verify that there exists a neighborhood V of x in M such that V ⊆ U and V ∩ S is connected.If this fails to be the case, then for all positive integer n large enough sothat B ( x, /n ) is convex and B ( x, /n ) ⊆ U , the set S n := S ∩ B ( x, /n ) isnot connected. Suppose that A n is the connected component of S n contains x , the set B n is another connected component of S n and y n is an arbitrarypoint of B n . Let γ : [0 , → M be the unique minimizing geodesic joining x, y n and hence its image is entirely in B ( x, /n ). M. R. POURYAYEVALI AND H. RADMANESH
Note that P S ( γ ( t )) ∈ S ∩ B ( x, /n ) for every t ∈ [0 , d ( P S ( γ ( t )) , x ) ≤ d ( P S ( γ ( t )) , γ ( t )) + d ( γ ( t ) , x ) ≤ d ( y n , γ ( t )) + d ( γ ( t ) , x )= d ( y n , x ) < /n. We now claim that the image of γ on [0 ,
1] is not entirely in Unp( S ). Oth-erwise, the continuity of P S on Unp( S ) ([20, Lemma 4.11]) implies that theset P S ( γ ([0 , P S ( γ ([0 , ⊆ S n and contains x , wehave P S ( γ ([0 , ⊆ A n . It follows that y n ∈ A n which contradicts ourchoice of y n . Then there exists a sequence { z n } such that z n / ∈ Unp( S )and d ( x, z n ) < /n . It implies that reach( S, x ) = 0 and this contradictioncompletes the proof of the lemma. (cid:3) (cid:3)
Lemma 3.2 implies that every closed prox-regular subset of M is locallyconnected. Example 3.3.
A well known example of a connected set which is not locallyconnected is the comb space, C = ([0 , × ∪ ( K × [0 , ∪ (0 × [0 , , in R where K = { /n : n ∈ N } . Note that this set is not prox-regular,because for every x ∈ (0 × [0 , , reach( C, x ) = 0 . In the following theorem, we obtain a characterization of proximal normalcones to prox-regular subsets of M . Theorem 3.4.
Suppose that S is a closed subset of M with the property thatits boundary, denoted by ∂S , is an embedded k -dimensional submanifold of M and x ∈ ∂S . Then (a) If ∂S is C , then N PS ( x ) ⊆ T ⊥ x ∂S where T ⊥ x ∂S is the normal space to ∂S at x . (b) If in addition S is prox-regular with nonempty interior and ∂S is C ,then there exist a neighborhood U of x in M and a C submersion ψ : U → R such that U ∩ ∂S = ψ − (0) and proximal normal cone to S at x is one ofthe following N PS ( x ) = cone {∇ ψ ( x ) } , or N PS ( x ) = span {∇ ψ ( x ) } . Proof.
Since ∂S is an embedded k -dimensional submanifold of M , thereexists a neighborhood U of x in M such that U ∩ ∂S is a level set of asubmersion ψ : U → R n − k , ψ = ( ψ , · · · , ψ n − k ). If ∂S is C , then along thesame lines as the proof of [6, Proposition 1.9], we have N PS ( x ) ⊆ N P∂S ( x ) ⊆ span {∇ ψ i ( x ) : i = 1 , · · · , n − k } = T ⊥ x ∂S. If in addition S is prox-regular with nonempty interior and ∂S is C , then ∂S is a codimension 1 submanifold of M . Moreover, by Lemma 3.2, S islocally connected and hence by shrinking U if necessary, we may assume INIMIZING CURVES IN PROX-REGULAR SETS 7 that U is convex and U ∩ S is connected. If U ∩ S ◦ = ∅ where S ◦ denotesthe interior of S (or there exists a neighborhood V ⊆ U of x such that V ∩ S ◦ = ∅ ), then U ∩ S = U ∩ ∂S and by [6, Proposition 1.9], we have N PS ( x ) = N P∂S ( x ) = span {∇ ψ ( x ) } . Now let U ∩ S ◦ be nonempty. Since U ∩ S ◦ is connected and U ∩ ∂S = ψ − (0), we have ψ ( U ∩ S ◦ ) ⊆ ( −∞ ,
0) or ψ ( U ∩ S ◦ ) ⊆ (0 , + ∞ ) . Replacing ψ by − ψ if necessary, we can assume that ψ ( y ) ≤ y ∈ U ∩ S . Let ξ := λ ∇ ψ ( x ) for some λ ≥
0. For given σ >
0, we define h ( y ) := (cid:10) − ξ, exp − x y (cid:11) + σd ( x, y ) + λψ ( y ) , for every y ∈ U . Then ∇ h ( x ) = 0 and for σ sufficiently large, Hess h ( x ) ispositive definite because for every v ∈ T x M we haveHess h ( x )( v ) = d dt | t =0 ( h (exp x ( tv )))= d dt | t =0 (cid:0) h− ξ, tv i + σt k v k + λψ (exp x ( tv )) (cid:1) = 2 σ k v k + λ Hess ψ ( x )( v ) . Therefore h has a local minimum at x and so there exists a neighborhood V of x such that V ⊆ U and for every y ∈ V ∩ S we have (cid:10) ξ, exp − x y (cid:11) ≤ σd ( x, y ) + λψ ( y ) ≤ σd ( x, y ) . It follows that ξ ∈ N PS ( x ) which completes the proof of the theorem. (cid:3) (cid:3) It is worth mentioning that in part (b) of Theorem 3.4 if the interior of S is empty, then S = ∂S and by [6, Proposition 1.9] we have N PS ( x ) = span {∇ ψ i ( x ) : i = 1 , · · · , n − k } = T ⊥ x ∂S. Example 3.5.
Let S be the set ( { } ∪ [1 , + ∞ )) × R in R and consider thepoints (0 , , (1 , ∈ ∂S . Then S is prox-regular and has a smooth boundary.At the point (0 , we have ψ ( x, y ) = x , N PS (0 ,
0) = span { (1 , } and at thepoint (1 , , ψ ( x, y ) = 1 − x and N PS (1 ,
0) = cone { ( − , } . In what follows, the closed set S c ∪ ∂S is denoted by ˆ S . Note that ∂ ˆ S ⊆ ∂S and if the point x ∈ ∂S is such that x / ∈ ∂ ˆ S , then x is the interior point ofˆ S . Theorem 3.6.
Suppose that S is prox-regular and ∂S is a C submanifoldof M . If x ∈ ∂S , then (3.1) T BS ( x ) ∩ T B ˆ S ( x ) = T x ∂S. Proof.
In the case when S ◦ = ∅ , we have T BS ( x ) = T x ∂S and T B ˆ S ( x ) = T x M .So we assume that the interior of S is nonempty. Let U and the submersion ψ : U → R be the ones applied in the proof of Theorem 3.4. If U ∩ S ◦ = ∅ M. R. POURYAYEVALI AND H. RADMANESH (or there exists a neighborhood V ⊆ U of x such that V ∩ S ◦ = ∅ ), then U ⊆ ˆ S and T B ˆ S ( x ) = T x M . Hence the expression (3.1) holds.We now consider the case in which U ∩ S ◦ is nonempty and for everyneighborhood V of x contained in U , V ∩ S ◦ = ∅ . Then U ∩ ∂ ˆ S = U ∩ ∂S and we claim that N P ˆ S ( x ) = cone {−∇ ψ ( x ) } . Indeed, Since Unp ( ∂S ) ⊆ Unp (cid:16) ˆ S (cid:17) and ∂S is a C submanifold of M , forevery z ∈ ∂ ˆ S ⊆ ∂S we havereach( ˆ S, z ) ≥ reach( ∂S, z ) > . Then reach( ˆ
S, z ) > z ∈ ˆ S and by Lemma 3.2, ˆ S is locallyconnected. Without loss of generality, we assume that U ∩ ˆ S is connected.By the choice of ψ we have ψ ( y ) ≤ y ∈ U ∩ S . On the other hand, ψ is a submersion on U and U ∩ ∂ ˆ S = U ∩ ∂S = ψ − (0). Then ψ ( y ) ≥ y ∈ U ∩ ˆ S and the claim is proved by a procedure similar to the proofof Theorem 3.4. So we have N PS ( x ) ∪ N P ˆ S ( x ) = T ⊥ x ∂S and N PS ( x ) ∩ N P ˆ S ( x ) = { } . Let us now prove the expression (3.1). Since U ∩ ∂ ˆ S = U ∩ ∂S , theset ˆ S is prox-regular and applying Lemma 3.1, we deduce that T x ∂S ⊆ T BS ( x ) ∩ T B ˆ S ( x ). Let v ∈ T BS ( x ) ∩ T B ˆ S ( x ) and w ∈ T ⊥ x ∂S be arbitrary.Without loss of generality, we assume that w ∈ N PS ( x ). Thus − w ∈ N P ˆ S ( x )and applying Lemma 3.1, we have h v, w i ≤ h v, − w i ≤
0. It followsthat v ∈ T x ∂S . (cid:3) (cid:3) Theorem 3.7.
Let S be a ϕ -convex subset of M , x ∈ S and U be a convexneighborhood of x . Then d (cid:0) exp − x y, T BS ( x ) (cid:1) ≤ ϕ ( x ) d ( x, y ) , for every y ∈ U ∩ S .Proof. Since T BS ( x ) is a closed convex subset of T x M , for y ∈ U ∩ S thereexists a unique vector v ∈ T BS ( x ) such that d (cid:0) exp − x y, T BS ( x ) (cid:1) = k exp − x y − v k . Therefore we have exp − x y − v ∈ N PT BS ( x ) ( v ) . Let us now show that N PT BS ( x ) ( v ) ⊆ N PS ( x ). Clearly, h ξ, v i = 0 for every ξ ∈ N PT BS ( x ) ( v ). Let ξ ∈ N PT BS ( x ) ( v ), then for every w ∈ T BS ( x ), h ξ, w i = h ξ, w − v i + h ξ, v i ≤ . Thus ξ ∈ (cid:0) T BS ( x ) (cid:1) ◦ and by Lemma 3.1 it follows that ξ ∈ N PS ( x ) . INIMIZING CURVES IN PROX-REGULAR SETS 9
Hence exp − x y − v ∈ N PS ( x ) and so we have (cid:28) exp − x y − v k exp − x y − v k , exp − x y (cid:29) ≤ ϕ ( x ) d ( x, y ) . This implies that k exp − x y − v k ≤ ϕ ( x ) d ( x, y ) which completes the proof. (cid:3)(cid:3) We are now ready to prove that the projection map P S is locally Lipschitzon an open set containing S , where S is a prox-regular subset of M . In [13],this property of prox-regular sets is verified in the special case in which M is a Hadamard manifold.Recall that the Hessian of a C function ψ on M is defined byHess ψ ( x )( v, w ) := h∇ X ∇ ψ, Y i ( x ) , for every x ∈ M and v, w ∈ T x M where X , Y are any vector fields such that X ( x ) = v and Y ( x ) = w and ∇ ψ denotes the gradient of ψ . Lemma 3.8.
Let M be a Riemannian manifold and x ∈ M . Assume that R > and k > are given such that | k |≤ k for every sectional curvature k on B ( x, R ) . Then the function ψ ( z ) := d ( x, z ) is smooth on B ( x, r ) forevery r > with r < min n r ( x ) , R, π √ k o and Hess ψ ( z )( w ) ≥ c ( z ) k w k , for every z ∈ B ( x, r ) and w ∈ T z M , where c ( z ) = min n , p k d ( x, z ) cot (cid:16)p k d ( x, z ) (cid:17)o . Proof.
Let z ∈ B ( x, r ) and w ∈ T z M . Thus according to the proof of [1,Proposition 2.2], we haveHess ψ ( z )( w ) = 2 l h D t X ( l ) , X ( l ) i , where l = d ( x, z ), X is the unique Jacobi field along γ with the propertythat X (0) = 0 and X ( l ) = w and γ is the unique minimizing geodesic,parameterized by arc length, such that γ (0) = x and γ ( l ) = z .Let w = w ⊤ + w ⊥ be the orthogonal decomposition of w where w ⊤ istangent to γ and w ⊥ is orthogonal to ˙ γ at z . Using Propositions 2.3 and2.4 of Chapter IX of [16], the Jacobi field X can be decomposed into X = X ⊤ + X ⊥ where X ⊤ and X ⊥ are Jacobi fields along γ with the propertythat X ⊤ and D t X ⊤ are tangent to γ and X ⊥ and D t X ⊥ are orthogonal to γ . So X ⊤ ( l ) = w ⊤ and X ⊥ ( l ) = w ⊥ and using the proof of [1, Proposition ψ ( z )( w ) = 2 l h D t X ( l ) , X ( l ) i = 2 l (cid:10) D t X ⊤ ( l ) , X ⊤ ( l ) (cid:11) + 2 l (cid:10) D t X ⊥ ( l ) , X ⊥ ( l ) (cid:11) ≥ l (cid:16) l k w ⊤ k (cid:17) + 2 l √ k cot (cid:0) l √ k (cid:1) k w ⊥ k = 2 k w ⊤ k + 2 l √ k cot (cid:0) l √ k (cid:1) k w ⊥ k ≥ c ( z ) k w k where c ( z ) = min (cid:8) , √ k d ( x, z ) cot (cid:0) √ k d ( x, z ) (cid:1)(cid:9) . (cid:3)(cid:3) Theorem 3.9.
Suppose that S is a closed prox-regular subset of a Riemann-ian manifold M . Then P S is locally Lipschitz on an open set V containing S .Proof. Since S is prox-regular, there exists a continuous function ϕ : S → [0 , ∞ ) such that S is ϕ -convex. Let x ∈ S and R >
R < r ( x )and B ( x, R ) has compact closure and B ( x, R ) ⊆ Unp( S ). Suppose that k > ρ > | k |≤ k for every sectionalcurvature k on B ( x, R ) and ϕ ( z ) ≤ ρ for every z ∈ B ( x, r ) ∩ S . Consider¯ r > r ≤ r ( z ) for every z ∈ B ( x, R ).Let a ∈ R be the solution of the equation 2 t cot( t ) = 1 on the interval (cid:0) , π (cid:1) . So we have 2 t cot( t ) > t ∈ (0 , a ). We now choose r > r < min (cid:26) R , ¯ r, ρ , a √ k (cid:27) . We show that P S is Lipschitz on B ( x, r ). To this end, let x , x ∈ B ( x, r ).The case x , x ∈ S is trivial, hence we suppose that x / ∈ S . Similar to theproof of [13, Theorem 3.13], we consider the following optimization problems(3.2) min s ∈ S ∩ B ( x,R ) d ( x , s ) = min v ∈ exp − x ( S ∩ B ( x,R )) d ( x , exp x v ) , (3.3) min s ∈ S ∩ B ( x,R ) d ( x , s ) = min v ∈ exp − x ( S ∩ B ( x,R )) d ( x , exp x v ) . Let P S ( x ) = s and P S ( x ) = s , hence s ∈ B ( x, R ) and s is the optimalsolution of (4.1). Moreover, s ∈ B ( x , r ) ⊆ B ( x, R ) because x ∈ S and d ( x , s ) ≤ d ( x , x ) < r .We claim that there exists a positive constant σ such that d ( x , s ) ≥ d ( x , s ) + σd ( s, s ) , for every s ∈ S ∩ B ( x , r ).Let s ∈ S ∩ B ( x , r ) and γ ( t ) = exp s (cid:0) t exp − s s (cid:1) be the unique geo-desic joining s and s which is entirely in B ( x , r ). We now define ψ ( z ) := d ( x , z ) for every z ∈ M . Then using the Taylor expansion, there exists t ∈ (0 ,
1) such that(3.4) d ( x , s ) = d ( x , s ) − h exp − s x , exp − s s i + 12 Hess ψ ( x )( v ) , INIMIZING CURVES IN PROX-REGULAR SETS 11 where x = γ ( t ) and v = ˙ γ ( t ). By Lemma 3.8,Hess ψ ( x )( v ) ≥ c ( x ) k v k = c ( x ) d ( s, s ) , where c ( x ) = min (cid:8) , √ k d ( x , x ) cot (cid:0) √ k d ( x , x ) (cid:1)(cid:9) . Since x ∈ B ( x , r ),by the choice of r we have2 p k d ( x , x ) cot (cid:16)p k d ( x , x ) (cid:17) > , and so c ( x ) >
1. Moreover, exp − s x ∈ N PS ( s ), hence h exp − s x , exp − s s i ≤ ϕ ( s ) d ( x , s ) d ( s, s ) ≤ ρrd ( s, s ) . Therefore (3.4) turns into(3.5) d ( x , s ) ≥ d ( x , s ) + (cid:18) − ρr (cid:19) d ( s, s ) . We put σ = (cid:0) − ρr (cid:1) , hence our choice of r guarantees that σ > x ( w i ) = s i for i = 1 ,
2, then (3.5) implies that d ( x , exp x ( v )) ≥ d ( x , exp x ( w )) + σc k v − w k , for every v ∈ exp − x ( S ∩ B ( x , r )), where c is the Lipschitz constant ofexp − x on B ( x, R ).Let c be a Lipschitz constant of exp x on B (0 x , R ), then by Shapiro’svariational principle we finally get d ( P S ( x ) , P S ( x )) = d (exp x w , exp x w ) ≤ c k w − w k≤ κc c σ d ( x , x ) , where κ is a positive constant such that 2 κ d ( x , x ) is a Lipschitz constantof the function f ( v ) = d ( x , exp x ( v )) − d ( x , exp x ( v )) on the neighborhood W := exp − x ( B ( x , r )) of w . (cid:3) (cid:3) We recall that a continuous map r : X → A from a topological space X to a subspace A of X is said to be a retraction if the restriction to A of r is the identity map. A subset S of a Riemannian manifold M is called L -retract if there exist a neighborhood V of S , a retraction r : V → S anda positive constant L such that d ( x, r ( x )) ≤ Ld S ( x ) , ∀ x ∈ V. Proposition 3.10. If S is a prox-regular subset of M , then S is L -retractwith L = 1 .Proof. According to Theorem 3.9, the projection map P S : V → S is alocally Lipschitz retraction from a neighborhood V of S to S . (cid:3) (cid:3) Directional differentiability of the metric projection at aboundary point
In this section by applying Shapiro’s variational principle, we investigatethe directional differentiability of the projection map P S at the boundarypoints of S where S is a prox-regular subset of a Riemannian manifold M .Let us recall the definition of directional differentiability for maps betweentwo Riemannian manifolds. Definition 4.1.
Let f : M → N be a map between two Riemannian man-ifolds, x ∈ M and ( V, φ ) be a chart of N at the point f ( x ) . We define thedirectional derivative of f at x in the direction v ∈ T x M as f ′ ( x ; v ) := lim t → + φ ( f (exp x ( tv ))) − φ ( f ( x )) t , when the limit exists.Moreover, the map f is said to be directionally differentiable at x if thedirectional derivative f ′ ( x ; v ) exists for all v ∈ T x M . In fact, f ′ ( x ; v ) is the right-handed derivative of the curve γ ( t ) := f (exp x ( tv ))at t = 0. Theorem 4.2.
Let S be a prox-regular subset of M and x ∈ S . Then P S isdirectionally differentiable at x and for every v ∈ T x MP ′ S ( x ; v ) = P T BS ( x ) ( v ) , where P T BS ( x ) denotes the metric projection to T BS ( x ) .Proof. Prox-regularity of S implies the existence of a continuous function ϕ : S → [0 , ∞ ) such that S is ϕ -convex. Let B ( x, r ) ⊆ Unp( S ) be a convexball with compact closure and v ∈ T x M . We are going to show thatlim t → + exp − x ( P S (exp x ( tv ))) t = P T BS ( x ) ( v ) . Since T BS ( x ) is a closed convex cone in T x M , we have P T BS ( x ) ( tv ) = tP T BS ( x ) ( v ) ∀ t ≥ , and so equivalently we must prove thatlim t → + k exp − x ( P S (exp x ( tv ))) − P T BS ( x ) ( tv ) k t = 0 . This means that k exp − x ( P S (exp x ( tv ))) − P T BS ( x ) ( tv ) k = o ( t ) . To this end, let t > t < r k v k and consider the followingoptimization problems(4.1) min w ∈ T BS ( x ) k w − tv k and(4.2) min y ∈ S ∩ B ( x,r ) d ( y, exp x tv ) = min w ∈ exp − x ( S ∩ B ( x,r )) d (exp x w, exp x tv ) . Note that exp x tv ∈ B ( x, r ) ⊆ Unp( S ), then we get ¯ x = P S (exp x ( tv )).Since x ∈ S , we have d (¯ x, x ) ≤ d (¯ x, exp x ( tv )) + d (exp x ( tv ) , x ) ≤ d (exp x ( tv ) , x ) = 2 t k v k < r, so ¯ x ∈ B ( x, r ). Hence ¯ v = exp − x ( P S (exp x ( tv ))) is the optimal solution of(4.2).Let v ∗ be the optimal solution of (4.1), then v ∗ = P T BS ( x ) ( tv ). Furthermoreusing the proof of [22, Theorem 3.1], k w − tv k ≥ k v ∗ − tv k + k w − v ∗ k , and (2.3) is the case for α = 1. We take ¯ r := 2 t k v k and W := B (0 , ¯ r ) ⊆ T x M ,then ¯ v, v ∗ ∈ W and by Shapiro’s variational principle, k ¯ v − v ∗ k ≤ ϑ ( t ) , where ϑ ( t ) = κ ( t ) + 2 δ ( t ) + ( k ( t ) δ ( t ) + k ( t ) δ ( t )) / , and κ ( t ) is a Lipschitz constant of the function h t ( w ) := d (exp x w, exp x tv ) − k w − tv k , on W . Moreover, k ( t ) and k ( t ) are Lipschitz constants of the functions f t ( w ) := k w − tv k and g t ( w ) := d (exp x w, exp x tv ) on W , respectively and δ ( t ) = sup (cid:8) d (cid:0) exp − x y, T BS ( x ) (cid:1) : y ∈ S ∩ B ( x, ¯ r ) (cid:9) ,δ ( t ) = d (cid:0) v ∗ , exp − x ( S ∩ B ( x, ¯ r )) (cid:1) . We now show that ϑ ( t ) = o ( t ). Indeed, k ( t ) ≤ t k v k and k ( t ) = max w ∈ W kh− − x ( w ) exp x ( tv ) , d exp x ( w ) ik≤ max w ∈ W (2 d (exp x ( w ) , exp x ( tv )) k d exp x ( w ) k ) ≤ lt k v k , where l = max w ∈ W k d exp x ( w ) k . Hence k ( t ) → k ( t ) → t → + .Moreover, by Theorem 3.7, for every y ∈ S ∩ B ( x, r ) d (cid:0) exp − x y, T BS ( x ) (cid:1) ≤ ϕ ( x ) d ( x, y ) . So we have lim y → x y ∈ S d (cid:0) exp − x y, T BS ( x ) (cid:1) d ( x, y ) = 0 , and this implies that δ ( t ) = o ( t ). Also, δ ( t ) = o ( t ) since δ ( t ) = d (cid:0) tv , exp − x ( S ∩ B ( x, ¯ r )) (cid:1) ≤ c d (exp x ( tv ) , S ) , where v = P T BS ( x ) ( v ) and c is a Lipschitz constant of exp − x on B ( x, ¯ r ).It remains only to verify that κ ( t ) = o ( t ). Indeed, for every w ∈ B (0 , r )and z ∈ T x M , ∇ h t ( w )( z ) = − D exp − x ( w ) exp x ( tv ) , d exp x ( w ) z E − h w − tv, z i . For fixed w, z we define F ( t ) = D exp − x ( w ) exp x ( tv ) , d exp x ( w ) z E , for every t with | t | < r k v k . The Taylor expansion gives F ( t ) = F (0) + F ′ (0) t + o ( t ) ∀ t. The values F (0) and F ′ (0) is obtained as follows: according to [16, Lemma3.5, p. 250] we have F (0) = D exp − x ( w ) x, d exp x ( w ) z E = D d exp exp x ( w ) ( − ˙ γ (1)) (cid:16) exp − x ( w ) x (cid:17) , z E , where γ is the geodesic γ ( t ) = exp x ( tw ) and hence˙ γ (1) = − exp − x ( w ) x. For simplicity, let us write ¯ w = exp − x ( w ) x . Thus using [16, Theorem 3.1], d exp exp x ( w ) ( ¯ w )( ¯ w ) = J (1) , where J is the Jacobi field along the geodesic β joining exp x ( w ) , x satisfyingthe properties ˙ β (0) = ¯ w , J (0) = 0 and D t J (0) = ¯ w . In fact, β ( t ) = γ (1 − t )and J ( t ) = t ˙ β ( t ) and so J (1) = − w and F (0) = −h w, z i .Also we have F ′ (0) = D d exp − x ( w ) ( x ) v, d exp x ( w ) z E = D d exp exp x ( w ) (cid:16) exp − x ( w ) x (cid:17) (cid:16) d exp − x ( w ) ( x ) v (cid:17) , z E = D d (cid:16) exp exp x ( w ) o exp − x ( w ) (cid:17) ( x ) v, z E = h v, z i . It follows that ∇ h t ( w )( z ) = o ( t ) for every w ∈ B (0 , r ) and z ∈ T x M . Thisimplies that κ ( t ) = o ( t ). (cid:3) (cid:3) Using Theorem 4.2, we obtain the following characterization of Bouligandtangent cone to a prox-regular set.
Corollary 4.3.
Let S be a closed prox-regular subset of M and x ∈ S . Then v ∈ T BS ( x ) if and only if there exists a continuous curve α : [0 , ε ) → S suchthat α (0) = x and ˙ α (0 + ) = v , where ˙ α (0 + ) is the right-handed derivative of α at . INIMIZING CURVES IN PROX-REGULAR SETS 15
Proof.
Let v ∈ T BS ( x ). We choose ε > x ( tv ) ∈ Unp( S ) for all t ∈ [0 , ε ). We now define α ( t ) := P S (exp x ( tv )) ∀ t ∈ [0 , ε ) . Then by Theorem 4.2,˙ α (0 + ) = P ′ S ( x ; v ) = P T BS ( x ) ( v ) = v. The proof of the converse statement is straightforward. (cid:3) (cid:3) Minimizing curves in prox-regular sets
Our goal in this section is to derive a necessary condition for a curve γ to be a minimizing curve between its endpoints in a prox-regular set. Tothis end, we employ the first variation formula. Let S ⊆ M be a closedprox-regular set whose boundary is a C Riemannian submanifold of M .In this situation, a continuous map γ : [ a, b ] → M is called a piecewiseregular curve if it is a piecewise C curve with nonzero derivatives. Moreover,by an admissible curve we mean a piecewise regular curve γ : [ a, b ] → M which is entirely in S . An admissible curve γ in S is said to be minimizingif L ( γ ) ≤ L (˜ γ ) for all admissible curves ˜ γ with the same endpoints where L ( γ ) denotes the length of γ in M .An admissible family of curves in S is a continuous map Γ : [0 , ε ) × [ a, b ] → M with the property that Γ( s, t ) ∈ S for all ( s, t ) ∈ [0 , ε ) × [ a, b ] and thereexists a partition a = a < · · · < a k = b of [ a, b ] such that Γ | [0 ,ε ) × [ a i − ,a i ] is C for every i = 1 , . . . , k . A variation of an admissible curve γ : [ a, b ] → M is an admissible family Γ in S such that Γ(0 , t ) = γ ( t ) for all t ∈ [ a, b ] andif in addition Γ( s, a ) = γ ( a ) and Γ( s, b ) = γ ( b ) for all s ∈ [0 , ε ), then it iscalled a proper variation.Recall that if Γ is a variation of γ , then the piecewise C vector field V along γ defined by V ( t ) = dds | s =0 + Γ( s, t ) is called the variation field of Γ,where dds | s =0 + denotes the right-handed derivative of Γ( ., t ) : [0 , ε ) → M at s = 0. Note that according to Corollary 4.3, if V is the variation field of avariation along γ , then V ( t ) ∈ T BS ( γ ( t )) ∀ t ∈ [ a, b ] . In the following, we investigate when a vector field along an admissiblecurve γ is the variation field of a variation of γ . Lemma 5.1.
Suppose that the closed set S is prox-regular and ∂S is a C submanifold of M . If x ∈ ∂S and v ∈ (cid:0) T BS ( x ) \ T x ∂S (cid:1) ∪ { } , then thereexists ε > such that exp x ( tv ) ∈ S for all t ∈ [0 , ε ) .Proof. Assuming the contrary, there exists a sequence { t n } such that t n ↓ x ( t n v ) ∈ S c ⊆ ˆ S . Moreover, v = lim n →∞ exp − x (exp x ( t n v )) t n . Hence v ∈ T B ˆ S ( x ) and so by Theorem 3.6, we have v ∈ T x ∂S . This contra-diction completes the proof. (cid:3) (cid:3) Lemma 5.2.
Suppose that γ : [ a, b ] → M is an admissible curve and V isa piecewise C vector field along γ . If for any t ∈ [ a, b ] with γ ( t ) ∈ ∂S wehave V ( t ) ∈ (cid:0) T BS ( γ ( t )) \ T γ ( t ) ∂S (cid:1) ∪ { } , then V is the variation field of a variation Γ of γ .Proof. Lemma 5.1 along with the compactness of [ a, b ] imply that thereexists ε > , ε ) × [ a, b ] → M defined by Γ( s, t ) :=exp γ ( t ) ( sV ( t )) is the desired variation of γ in S . (cid:3) (cid:3) The following theorem gives a necessary condition for a curve to be min-imizing in S . Theorem 5.3.
Let γ : [ a, b ] → M be a unit speed admissible curve. If γ isminimizing in S , then (5.1) D t ˙ γ ( t ) ∈ N PS ( γ ( t )) , for every t ∈ [ a, b ] except for finitely many points.Proof. If the interior of S is empty, then (5.1) evidently holds, since in thiscase N PS ( x ) = T ⊥ x ∂S for every x ∈ S . Therefore we assume that the interiorof S is nonempty.Let a = a < a < · · · < a k = b be a partition of [ a, b ] such that γ is C oneach subinterval [ a i − , a i ] and t ∈ [ a, b ] be such that t = a i for each i . Wesuppose that t ∈ ( a j − , a j ) for some j , 1 ≤ j ≤ k and we get x := γ ( t ). If x ∈ S ◦ , then there exist an open neighborhood U of x in M and a positivenumber δ such that γ ( t ) ∈ U ⊆ S for all t ∈ I := [ t − δ, t + δ ]. Hence γ | I is minimizing in M and this implies that D t ˙ γ ( t ) = 0.Assume that x ∈ ∂S and let the open neighborhood U and the submer-sion ψ : U → R be the ones applied in the proof of Theorem 3.4. Clearly D t ˙ γ ( t ) = 0 or there is a positive number ǫ such that γ ( t ) ∈ U ∩ ∂S for all t ∈ I := [ t − ǫ, t + ǫ ] ⊂ ( a j − , a j ). So it suffices to check that (5.1) holdsin the latter case. Indeed, γ | I is minimizing in the Riemannian submanifold ∂S of M . Then we have(5.2) D t ˙ γ ( t ) ∈ T ⊥ γ ( t ) ∂S ∀ t ∈ I. If U ∩ S ◦ = ∅ (or there exists a neighborhood V ⊆ U of x such that V ∩ S ◦ = ∅ ), then N PS ( x ) = T ⊥ x ∂S . Otherwise, in order to deduce that D t ˙ γ ( t ) ∈ N PS ( γ ( t )), by Lemma 3.1 it suffices to show that h D t ˙ γ ( t ) , v i ≤ ∀ v ∈ T BS ( γ ( t )) . If this fails to hold, then there is η ∈ T BS ( γ ( t )) such that h D t ˙ γ ( t ) , η i > . Thus the inclusion (5.2) implies that η / ∈ T γ ( t ) ∂S . INIMIZING CURVES IN PROX-REGULAR SETS 17
We now construct a vector field along γ such that for any t ∈ [ a, b ] with γ ( t ) ∈ ∂S , V ( t ) ∈ (cid:0) T BS ( γ ( t )) \ T γ ( t ) ∂S (cid:1) ∪ { } . We define V ( t ) := L γt t η and g ( t ) := (cid:10) V ( t ) , ∇ ψ ( γ ( t )) (cid:11) for all t ∈ I . Ac-cording to Theorem 3.4, N PS ( γ ( t )) = cone {∇ ψ ( γ ( t )) } for all t ∈ I . Then g ( t ) < g implies that g ( t ) < t . It follows that V ( t ) ∈ T BS ( γ ( t )) \ T γ ( t ) ∂S ∀ t ∈ I, without loss of generality. By shrinking I if necessary, we can assume that(5.3) (cid:10) V ( t ) , D t ˙ γ ( t ) (cid:11) > ∀ t ∈ I. We choose a bump function φ ∈ C ∞ ( R ) with support in I such that φ ( t ) ≡ c , c ], where c , c is such that t − ǫ < c < t < c < t + ǫ . We nowdefine V ( t ) := φ ( t ) V ( t ) for all t ∈ [ a, b ]. Then V is the desired vector fieldalong γ .Applying Lemma 5.2 to the vector field V along γ gives rise to a variationΓ : [0 , ε ) × [ a, b ] → M of γ in S such that V is its variation field. Since γ is minimizing in S , for all s ∈ [0 , ε ) we have L (Γ s ) ≥ L (Γ ) where Γ s isan admissible curve on [ a, b ] defined by Γ s ( t ) := Γ( s, t ). This implies that dds | s =0 + L (Γ s ) ≥
0. Using the first variation formula (see [17, Theorem 6.3])we conclude that Z ba h V ( t ) , D t ˙ γ ( t ) i dt + k − X i =1 h V ( a i ) , ∆ i ˙ γ i ≤ , where ∆ i ˙ γ := ˙ γ (cid:0) a + i (cid:1) − ˙ γ (cid:0) a − i (cid:1) . Since V ( a i ) = 0 for each i = 1 , . . . , k , wehave Z ba h V ( t ) , D t ˙ γ ( t ) i dt ≤ . On the other hand, Z ba h V ( t ) , D t ˙ γ ( t ) i dt ≥ Z c c (cid:10) V ( t ) , D t ˙ γ ( t ) (cid:11) dt > , a contradiction which establishes that D t ˙ γ ( t ) ∈ N PS ( γ ( t )) . (cid:3) (cid:3) Example 5.4.
Let S be the -sphere of radius one in R with the roundmetric g ◦ which is induced from the Euclidean metric on R . Considerspherical coordinates ( θ, φ ) on the subset S − { ( x, y, z ) : x ≤ , y = 0 } ofthe sphere defined by ( x, y, z ) = (sin θ cos φ, sin θ sin φ, cos θ ) 0 < θ < π, − π < φ < π. It is known that the round metric is g ◦ = dθ + sin θ dφ in sphericalcoordinates. Also, Christoffel symbols of g ◦ in spherical coordinates are Γ θij = (cid:18) − sin θ cos θ (cid:19) , Γ φij = (cid:18) cos θ sin θ cos θ sin θ (cid:19) . Let S be the closed subset of S which is obtained by removing the sector θ < θ ≤ π from S where π < θ < π . According to [20, Theorem 4.18] , S is a prox-regular subset of S .Note that for every p ∈ ∂S , the map ψ : U → R defined by ψ ( θ, φ ) = θ − θ is the desired submersion which is used in Theorem 3.4. Hence applyingTheorem 3.4, we obtain N PS ( p ) = cone { (1 , } = { λ ∂/∂θ : λ ≥ } . Clearly, the unit speed curve γ defined by γ ( t ) := (cid:18) θ , t sin θ (cid:19) ∀ t ∈ I := [ − π/ θ , π/ θ ] , is a minimizing curve in S joining ( θ , − π/ and ( θ , π/ . It can be foundthat the curve γ satisfies the necessary condition (5.1). Indeed, we have D t ˙ γ ( t ) = (cid:18) − cos θ sin θ (cid:19) ∂∂θ ∀ t ∈ I. Then putting λ := − cos θ / sin θ , we observe that λ > and (5.1) holds.On the other hand, consider another admissible curve α in S joining ( θ , − π/ and ( θ , π/ defined by α ( t ) := ( θ − t, − π/
2) 0 ≤ t ≤ θ − π/ π/ , t − θ ) θ − π/ ≤ t ≤ θ + π/ t − θ , π/ θ + π/ ≤ t ≤ θ . Note that D t ˙ α ( t ) = 0 and α satisfies (5.1), but it is not a minimizing curvein S , since L ( γ ) = π sin θ < π < L ( α ) . Example 5.5.
Let H be the hyperbolic plane; that is, the upper half-planein R with the metric g H = (cid:0) dx + dy (cid:1) /y . The Riemannian distancebetween two points z = ( x , y ) , z = ( x , y ) of H is as d ( z , z ) = 2 ln q ( x − x ) + ( y − y ) + q ( x − x ) + ( y + y ) √ y y . We consider the subset S = { ( x, y ) : 1 ≤ y ≤ } of H . It is evident that S is not convex in the Hadamard manifold (cid:0) H , g H (cid:1) . The metric projection P S is obtained as follows: P S ( x, y ) = (cid:26) ( x, if < y < x, if y > . Since for instance in the case when y > , for every s = x we have d (( x, y ) , ( s, > ln y d (( x, y ) , ( x, . INIMIZING CURVES IN PROX-REGULAR SETS 19
Then by [20, Corollary 4.20] , S is a prox-regular subset of H .Note that for every z = ( s, ∈ ∂S , the map ψ : H → R defined by ψ ( x, y ) = y − is the desired submersion which is needed in Theorem 3.4.Hence N PS ( z ) = cone { ∂/∂y } = { λ ∂/∂y : λ ≥ } . We now consider the unit speed curve γ in S defined by γ ( t ) := (2 t, ∀ t ∈ R . So D t ˙ γ ( t ) = 2 ∂∂y for all t ∈ R and it follows that the curve γ satisfies thenecessary condition (5.1). References [1] Azagra, D., Fry, R.: A second order smooth variational principle on Riemannianmanifolds. Canad. J. Math. 62, 241-260 (2010)[2] Bangert, V.: Sets with positive reach. Arch. Math. 38, 54-57 (1982)[3] Barani, A., Hosseini, S., Pouryayevali, M.R.: On the metric projection onto ϕ -convexsubsets of Hadamard manifolds. Rev. Mat. Complut. 26, 815-826 (2013)[4] Canino, A.: Local properties of geodesics on p -convex sets. Ann. Mat. Pura Appl.159(1), 17–44 (1991)[5] Canino, A.: On p -convex sets and geodesics. J. Differential Equations. 75, 118-157(1988)[6] Clarke, F.H., Ledyaev, Yu.S., Stern, R.J., Wolenski, P.R.: Nonsmooth Analysis andControl Theory. Graduate Texts in Mathematics 178, Springer, New York (1998)[7] Colombo, G., Thibault, L.: Prox-regular sets and applications. Handbook of Noncon-vex Analysis and Applications, D.Y. Gao and D. Motreanu Eds., International Press,Boston, 99-182 (2010)[8] Degiovanni, M., Marino, A., Tosques, M.: General properties of ( p, q )-convex functionsand ( p, q )-monotone operators. Ricerche Mat. 32, 285-319 (1983)[9] do Carmo, M.P.: Riemannian Geometry. Birkh¨auser, Boston (1992)[10] Federer, H.: Curvature measure. Trans. Amer. Math. Soc. 93, 418-491 (1959)[11] Gilbarg, D., Trudinger, N.S.: Elliptic Partial Differential Equations of Second Order.Springer-Verlag, Berlin, New York (1998)[12] Greene, R.E., Shiohama, K.: Convex functions on complete noncompact manifolds:topological structure. Invent. Math. 63, 129-157 (1981)[13] Hosseini, S., Pouryayevali, M.R.: On the metric projection onto prox-regular subsetsof Riemannian manifolds. Proc. Amer. Math. Soc. 141, 233-244 (2013)[14] Kleinjohann, N.: Convexity and the unique footpoint property in Riemannian geom-etry. Arch. Math. 35, 574-582 (1980)[15] Kruskal, J.: Two convex counterexamples: a discontinuous envelope function and anon-differentiable nearest-point mapping. Proc. Amer. Math. Soc. 23, 697-703 (1969)[16] Lang, S.: Fundamentals of Differential Geometry. Graduate Texts in Mathematics191, Springer-Verlag, New York (1999)[17] Lee, J.M.: Introduction to Riemannian Manifolds. Graduate Texts in Mathematics176, Springer, New York (2018)[18] Leobacher, G., Steinicke, A.: Existence, uniqueness and regularity of the projectiononto differentiable manifolds. arXiv preprint arXiv:1811.10578 (2018)[19] Poliquin, R.A., Rockafellar, R.T.: Prox-regular functions in variational analysis.Trans. Amer. Math. Soc. 348, 1805-1838 (1996)[20] Pouryayevali, M. R., Radmanesh, H.: Sets with the unique footpoint property and ϕ -convex subsets of Riemannian manifolds. J. Convex Anal. 26, 617-633 (2019) [21] Sakai, T.: Riemannian Geometry. Translations of Mathematical Monographs 149.American Mathematical Society (1996)[22] Shapiro, A.: Existence and differentiability of metric projections in Hilbert spaces.SIAM J. Optim. 4(1), 130-141 (1994)[23] Walter, R.: On the metric projection onto convex sets in Riemannian spaces. Arch.Math. (Basel), 25, 91-98 (1974) Department of Pure Mathematics, Faculty of Mathematics and Statistics,University of Isfahan, Isfahan, 81746-73441, Iran
Email address : [email protected] Email address ::