On the Cardinality of Positively Linearly Independent Sets
aa r X i v : . [ m a t h . O C ] S e p Optimization Letters manuscript No. (will be inserted by the editor)
On the Cardinality of Positively Linearly Independent Sets
W. Hare · H. Song
Received: date / Accepted: date
Abstract
Positive bases, which play a key role in understanding derivative free op-timization methods that use a direct search framework, are positive spanning setsthat are positively linearly independent. The cardinality of a positive basis in IR n hasbeen established to be between n +1 and n (with both extremes existing). The lowerbound is immediate from being a positive spanning set, while the upper bound uses both positive spanning and positively linearly independent. In this note, we providedetails proving that a positively linearly independent set in IR n for n ∈ { , } has atmost n elements, but a positively linearly independent set in IR n for n ≥ can havean arbitrary number of elements. Keywords:
Positive Linear Independence, Positive Basis, Convex Analysis
First introduced in 1954 [Dav54], the notion of a positive basis plays a key role in un-derstanding derivative free optimization methods that use a direct search framework(a.k.a. pattern search methods) [LT96,CP02,KLT03,Reg15]. In loose terms, a pos-itive basis is a set that provides some directional information into every half-space,without including unnecessary vectors.Pattern search methods seek a minimizer of f by evaluating f over an ever shrink-ing ‘pattern’:Given f and a incumbent solution x k .– Evaluate f ( x k + αv i ) for each v i ∈ V = { v , v , ..., v m } . W. HareMathematics, University of British Columbia, 3333 University Way, Kelowna, BC, Canada.E-mail: [email protected]. SongMathematics, University of Calgary, 2500 University Drive NW, Calgary, AB, Canada.E-mail: [email protected] W. Hare, H. Song – If a lower function value is found, then update x k .– If no lower function value is found, then shrink α .If f ∈ C , ∇ f ( x k ) = 0 , and the pattern forms a positive basis, then at least one vectorin the pattern is a descent direction for f at x k . Thus, under these conditions, once α is sufficiently small, descent must occur. Formalization of this idea has been used toprove convergence in a number of direct search/pattern search methods [LT96,CP02,KLT03,AD04] (and many others).Mathematically, to define positive basis we begin with the positive span: given aset of vectors V = { v , v , ..., v m } ⊆ IR n , we define the positive span of V byp-span ( V ) = ( m X i =1 α i v i : α i ≥ for all i = 1 , , ..., m ) . We now define V to be a positive basis if it satisfies two propertiesi) [Positive Spanning] p-span ( V ) = IR n , andii) [Positive Linear Independence] v i / ∈ p-span ( V\{ v i } ) for all i = 1 , , ..., m .Unlike bases of IR n (which always have exactly n elements), a positive basis of IR n has between n + 1 and n elements. The lower bound of n + 1 comes from thefollowing fact. Proposition 1 [Dav54, Thm 3.7] If V = { v , v , ..., v m } is a positive spanning setof IR n , then V ⊖ = { v , v , ..., v m − } is a spanning set of IR n . Thus, to satisfy positive spanning condition, a set must contain at least n + 1 vectors.The upper bound on the cardinality of a positive basis is more subtle to obtain. Theoriginal work of Davis includes a proof of this fact [Dav54, Thm 6.7], but the deriva-tion is rather technical. Alternate proofs appear in the works of Shephard [She71] andAudet [Aud11].It is important to note that all three proofs use both positive spanning and positivelinear independence to provide the upper bound. As such, the property of positivelinear independence is less clear in its implications on cardinality. For example, anyset consisting of exactly one vector is always positive linearly independent – includ-ing the set V = { } . (Of course, any positive linearly independent set with or morevectors cannot contain the vector.) Based on a knowledge of positive bases, onemight conjecture that positively linearly independent sets have at most n elements.This is true in IR and IR , but false beyond that. In this note, we provide detailsproving that a positively linearly independent set in IR n for n ∈ { , } has at most n elements, but a positively linearly independent set in IR n for n ≥ can exceedthis bound. Proofs of the first two statements appear in Section 2. In Section 3 weprovide two examples. The first is a straightforward example of a positively linearlyindependent set V with |V| = (cid:18) n ⌊ n ⌋ (cid:19) This example requires no tools other than basic linear algebra. The second exampledevelops a positively linearly independent set in IR n with an arbitrary number of n the Cardinality of Positively Linearly Independent Sets 3 elements. This example requires a small knowledge of convex analysis, but is stillquite accessible to a general audience.We conclude the introduction with a lemma that allows us to simplify notation inall future proofs and examples. Lemma 1
Let V = { v , v , . . . , v m } ⊆ IR n with v i = 0 for all i . Then V is pos-itively linearly independent if and only if b V = { v k v k , v k v k , . . . , v m k v m k } is positivelylinearly independent.Proof. Note that v k = X i = k α i v i ⇐⇒ v k k v k k = X i = k β i v i k v i k , where α i ≥ and β i = α i k v i kk v k k ≥ for any i = k . Thus, V is positively linearlyindependent if and only if b V is positively linearly independent. IR and IR Proposition 2
The maximal cardinality of a positively linearly independent set in IR is 2.Proof. Suppose V = { v , v , v } . If ∈ V , then V is not positively linearly inde-pendent. If / ∈ V , then, by Lemma 1, without loss of generality v i ∈ {− , } foreach i . By the pigeon hole principle, at least two vectors i and j are the same, and V is not positively linearly independent. Thus, if V is positively linearly independent,then |V| ≤ . Proposition 3
The maximal cardinality of positively linearly independent sets in IR is 4.Proof. For eventual contradiction, suppose V = { v , v , . . . , v } ⊆ IR is positivelylinearly independent. Without loss of generality, assume k v i k = 1 for i = 1 , , ..., .Clearly, v = v i for any i = 1 . Moreover, if v = − v i for some i , then v k = − v for any k = i . As such, there are at least two vectors, say v and v , that are linearlyindependent. Examining, V \ { v , v } , and repeating the same arguments, we mayassume that v and v are also linearly independent. Therefore, { v , v } and { v , v } are both bases of IR .As { v , v } is a basis of IR , there exists α ∈ IR such that v = α v + α v .If α ≥ and α ≥ , then we have contradicted the positive linear independence of V . If α > and α ≤ , then α v = v − α v , which also contradicts positivelinear independence. Similarly, α > and α ≤ fails. Hence, α ≤ .Using the same arguments, and { v , v } is a basis of IR , there exists β ≤ and γ ≤ such that v = β v + βv and v = γ v + γ v . Substitution now yields v = ( γ α + γ β ) v + ( γ α + γ β ) v W. Hare, H. Song which shows that v ∈ p-span ( V\{ v } ) , so V is not positively linearly independent.On a final note for this section, it is clear that the maximum cardinalities derivedabove can be achieved, as constructing an example with |V| = 2 n is trivial. IR n for n ≥ Our first example provides a family of positively linear independent sets whose car-dinality grows exponentially with dimension.
Example 1 In IR n , let V be the set of vectors of all the nontrivial permutations with ⌊ n ⌋ n − ⌊ n ⌋ = ⌈ n ⌉ V = ( v ∈ { , } n : n X i =1 v i = j n k) , then |V| = (cid:0) n ⌊ n ⌋ (cid:1) , and V is positively linearly independent. Proof.
Suppose v k ∈ V and there exist nonnegative numbers α i such that v k = P i = k α i v i . As v k = 0 , there must exist some j with α j > . Since v k = v j , and v k , v j ∈ V , there must exist some index l with v kl = 0 and v jl = 1 . This yields thecontridiction v kl = X i = k α i v il ≥ α j v jl = α j > . Hence, V is positively linearly independent. The proof of the cardinality is trivial.Our second example requires some basic tools and definitions from convex anal-ysis. Definition 1
A set C ⊆ IR n is convex, if given any x, y ∈ C and any θ ∈ [0 , , thepoint z = θx + (1 − θ ) y ∈ C .A set C ⊆ IR n is strictly convex, if given any x, y ∈ C with x = y and any θ ∈ (0 , ,the point z = θx + (1 − θ ) y ∈ int( C ) . Definition 2
The convex hull of a set S , denoted conv( S ) , is the smallest convex setthat contains S . Example 2
The set B = { x ∈ IR n : k x k ≤ } is strictly convex. Example 3 [RW09, Thm 2.27] Let Y = { y , y , ..., y m } ⊆ C . Then conv( Y ) = ( x = m X i =1 α i y i : α i ≥ , m X i =1 α i = 1 ) . Combining definition 1 and example 3, we have the following classic lemma. n the Cardinality of Positively Linearly Independent Sets 5
Lemma 2
Suppose C is strictly convex. Let Y = { y , y , ...y m } ⊆ C and ¯ x ∈ conv( Y ) . Then either x = y i for some i or x ∈ int( C ) . We now prove the main result.
Theorem 1
For n ≥ a positively linearly independent set in IR n may contain anarbitrary number of vectors.Proof. We shall provide an example in IR . Extending to IR n can be trivially accom-plished by adding elements in the extra dimensions.Let S = { ( x, y ) : x + y = 1 } . Select any distinct m points in S , { ( x , y ) , ( x , y ) , . . . , ( x m , y m ) } ⊂ S, and define V = { v , v , . . . , v m } = { ( x , y , , ( x , y , , . . . , ( x m , y m , } . We claim that V is positively linear independent.Suppose there exists v k ∈ p-span ( V \{ v k } ) . By reordering, without loss of gen-erality, assume k = m , so v m = m − X i =1 α i v i , with α i ≥ . Noting that v i = 1 for all i , we must have v m = m − X i =1 α i v i = m − X i =1 α i . This implies ( x m , y m ) = m − X i =1 α i ( x i , y i ) , m − X i =1 α i = 1 , α i ≥ . I.e., ( x m , y m ) ∈ conv( { ( x , y ) , ( x , y ) , . . . , ( x m − , y m − ) } ) . As ( x m , y m ) =( x i , y i ) for any i < m , Fact 2 implies ( x m , y m ) ∈ int( B ) , contradicting ( x i , y i ) ∈ B \ int( B ) for all i .It is interesting to note that the example in the proof of Theorem 1 does notactually require m to be finite. Indeed, in IR it is possible to create an uncountable setof vectors V such that v / ∈ p-span ( V\{ v } ) for all v ∈ V . Moreover, while the exampleuses the unit sphere to create the set, it is clear that S = { ( x, y ) : x + y = 1 } couldbe replaced by the boundary of any compact strictly convex set.Finally, we note that an alternate (but similar) proof of Theorem 1 was indepen-dently developed by Regis and presented in [Reg15, Thm 3.4]. W. Hare, H. Song
Acknowledgements
This research was partially funded by the Natural Sciences and Engineering ResearchCouncil (NSERC) of Canada, Discover Grant
References
AD04. C. Audet and J. Dennis. Mesh adaptive direct search algorithms for constrained optimization.
SIAM J. Optim. , 17:2006, 2004.Aud11. C. Audet. A short proof on the cardinality of maximal positive bases.
Optim. Lett. , 5(1):191–194,2011.CP02. I.D. Coope and C.J. Price. Positive bases in numerical optimization.
Comput. Optim. Appl. ,21(2):169–175, 2002.Dav54. C. Davis. Theory of positive linear dependence.
Amer. J. Math. , 76:733–746, 1954.KLT03. T.G. Kolda, R.M. Lewis, and V. Torczon. Optimization by direct search: new perspectives onsome classical and modern methods.
SIAM Rev. , 45(3):385–482, 2003.LT96. R.M. Lewis and V. Torczon. Rank ordering and positive bases in pattern search algorithms.Technical report, Institute for Computer Applications in Science and Engineering (ICASE), 1996.RW09. R.T. Rockafellar and R.J.-B Wets.
Variational Analysis . Grundlehren der Mathematischen Wis-senschaften [Fundamental Principles of Mathematical Sciences], 317. Springer-Verlag BerlinHeidelberg 1998, Corrected 3rd printing, 2009.Reg15. R.G. Regis. On the properties of positive spanning sets and pos-itive bases.
Optimization and Engineering , online first, 2015. http://link.springer.com/article/10.1007/s11081-015-9286-x
She71. G.C. Shephard. Diagrams for positive bases.