Restricted Linear Constrained Minimization of quadratic functionals
aa r X i v : . [ m a t h . O C ] S e p Restricted Linear Constrained Minimization ofquadratic functionals
Dimitrios PappasDepartment of Statistics,Athens University of Economics and Business,76 Patission Str, 10434, Athens, Greece([email protected], [email protected])September 14, 2018
Abstract
In this work a linearly constrained minimization of a positive semidef-inite quadratic functional is examined. Our results are concerning infinitedimensional real Hilbert spaces, with a singular positive operator relatedto the functional, and considering as constraint a singular operator. Thedifference between the proposed minimization and previous work on thisproblem, is that it is considered for all vectors perpendicular to the kernelof the related operator or matrix.
Keywords : Quadratic functional, Constrained Optimization, Moore-Penrose in-verse , Restricted Optimization. : 47A05, 47N10, 15A09.
The quadratic programming problem with equality constraints is one of thebasic problems in optimization, in both the finite and the infinite dimensional1ase. The general problem is to locate from within a given subset of a vectorspace the particular vector which minimizes a given functional. In this case,the subset of vectors is defined by a set of linear constraint relations and thefunctional is quadratic. In a classical book of Optimization Theory by Luen-berger [7], various similar optimization problems are presented, for both finiteand infinite dimensions.In the field of applied mathematics, a strong interest is shown in applicationsof the generalized inverse of matrices or operators. Generalized inverses canbe used whenever a matrix/ operator is singular, in many fields of both com-putational and also theoretical aspects. An application of the Moore-Penroseinverse in the finite dimensional case, is the minimization of a positive definitequadratic functional under linear constraints, presented in Manherz and Hakimi[8].The problem studied in their work in the following: minimizef ( x ) = h x, Qx i + h p, x i + a, x ∈ S where S = { x : Ax = b } and Q is a positive definite matrix.In this work we will extend this result for positive semidefinite matrices oroperators acting on infinite dimensional real Hilbert spaces. Since in this casethe operator studied is singular, the proposed minimization is attained for thevectors perpendicular to the kernel of the operator (or matrix) of the quadraticfunctional. The notion of the generalized inverse of a matrix was first introduced by H.Moore in 1920, and again by R. Penrose in 1955. These two definitions areequivalent and the generalized inverse of an operator or matrix is also called theMoore- Penrose inverse. It is known that when T is singular, then its uniquegeneralized inverse T † (known as the Moore- Penrose inverse) is defined. Inthe case when T is a real r × m matrix, Penrose showed that there is a unique2atrix satisfying the four Penrose equations, called the generalized inverse of T ,noted by T † . The generalized inverse, known as Moore-Penrose inverse, of anbounded linear operator T with closed range, is the unique operator satisfyingthe following four conditions: T T † = ( T T † ) ∗ , T † T = ( T † T ) ∗ , T T † T = T, T † T T † = T † (1)where T ∗ denotes the adjoint operator of T .In what follows, we consider H a separable infinite dimensional Hilbert space, B ( H ) denotes the set of all bounded operators on H and all operators mentionedare supposed to have closed range. In addition, R ( T ) will denote the range ofan operator T , and N ( T ) will denote its kernel.It is easy to see that R ( T † ) = N ( T ) ⊥ , T T † is the orthogonal projection of H onto R ( T ), denoted by P T , and that T † T is the orthogonal projection of H onto N ( T ) ⊥ = R ( T ∗ ) noted by P T ∗ . It is well known that R ( T † ) = R ( T ∗ ).It is also known that T † is bounded if and only if T has a closed range.If T has a closed range and commutes with T † , then T is called an EP operator.EP operators constitute a wide class of operators which includes the self adjointoperators, the normal operators and the invertible operators.Let us consider the equation T x = b, T ∈ B ( H ), where T is singular. If b / ∈ R ( T ),then the equation has no solution. Therefore, instead of trying to solve theequation k T x − b k = 0, we may look for a vector u that minimizes the norm k T x − b k . Note that the vector u is unique. In this case we consider the equation T x = P R ( T ) b , where P R ( T ) is the orthogonal projection on R ( T ).The following two propositions can be found in [5] and hold for operators andmatrices: Proposition 2.1
Let T ∈ R r × m and b ∈ R r , b / ∈ R ( T ) . Then, for u ∈ R m , thefollowing are equivalent:(i) T u = P R ( T ) b (ii) k T u − b k ≤k T x − b k , ∀ x ∈ R m iii) T ∗ T u = T ∗ b Let B = { u ∈ R m | T ∗ T u = T ∗ b } . This set of solutions is closed and convex,therefore, it has a unique vector with minimal norm. In the literature (eg.Groetsch [5]), B is known as the set of the generalized solutions. Proposition 2.2
Let T ∈ C r × m and b ∈ C r , b / ∈ R ( T ) , and the equation T x = b . Then, if T † is the generalized inverse of T , we have that T † b = u , where u isthe minimal norm solution defined above. This property has an application in the problem of minimizing a symmetricpositive definite quadratic functional subject to linear constraints, assumed con-sistent.As mentioned above, EP operators include normal and self adjoint operators,therefore the operator T in the quadratic form studied in this work is EP. Anoperator T with closed range is called EP if N ( T ) = N ( T ∗ ). It is easy to seethat T EP ⇔ R ( T ) = R ( T ∗ ) ⇔ R ( T ) ⊥ ⊕ N ( T ) = H ⇔
T T † = T † T. (2)We take advantage of the fact that EP operators have a simple canonical form T = U ( A ⊕ U ∗ according to the decomposition H = R ( T ) ⊕ N ( T ). Indeedan EP operator T has the following simple matrix form, T = A
00 0 , wherethe operator A : R ( T ) → R ( T ) is invertible, and its generalized inverse T † hasthe form T † = A −
00 0 (see [2], [4] ).In the finite dimensional case, if T ∈ R n × n and rank( T ) = r , the unitary matrix U has the form [ U , U ] where U ∈ R n × r is an orthonormal basis for the columnspace of T and U is an orthonormal basis for the null space of the matrix T .In Matzakos and Pappas [9] an algorithm is presented for the symbolic com-putation of the Moore-Penrose inverse for EP matrices and the correspondingfactorization T = U ( A ⊕ U ∗ is presented.As mentioned above, a necessary condition for the existance of a bounded gen-eralized inverse is that the operator has closed range. Nevertheless, the range of4he product of two operators with closed range is not always closed. In Izumino[6] an equivalent condition is given, using orthogonal projections : Proposition 2.3
Let A and B be operators with closed range. Then, AB hasclosed range if and only if A † ABB † has closed range. We will use the above theorem to prove the existence of the Moore- Penroseinverse used in our work.
The Generalized inverse of an operator or matrix plays a crucial role in manyoptimization problems where minimal norm solutions are studied. Ben-Israeland Greville ([1] chapter 3), and Campbell and Meyer ([2] chapter 3.6), haveconsidered the constrained least- squares problem minimize || Ax − b || under Bx = d with A, B, b, d, x all complex.In Chen [3], a similar problem is considered using partial ordering induced bycones, where the generalized inverse plays a fundamental role.The purpose of this paper is to extend the work done on this subject by Manherzand Hakimi [8]. At first we will extend the finite dimensional results in thecase of positive definite operators, and then we will examine the case when theoperator is singular.
Let Q be a positive definite symmetric matrix. The following theorem can befound in Manherz and Hakimi [8] : 5 heorem 3.1 Let Q be positive definite. Consider the equation Ax = b .If the set S = { x : Ax = b } is not empty, the the problem : minimize Φ( x ) = h x, Qx i + h p, x i + a, x ∈ S and p, a arbitrary has the unique solution x = Q − ( AQ − ) † ( 12 AQ − p + b ) − Q − p A generalization of the above theorem for infinite dimensional Hilbert spaces,is by replacing Q with an invertible positive operator T . The operator A mustbe singular, otherwise this problem is trivial. In the case of operators insteadof matrices, the proof is similar to Manherz and Hakimi [8], but the existenceof a bounded Moore- Penrose inverse is not trivial like in the finite dimensionalcase. Lemma 3.2
Let T ∈ B ( H ) be an invertible positive operator with closed rangeand A ∈ B ( H ) singular with closed range.Then, the range of AT − is closed , where T − denotes the operator ( T ) − .Proof. : Using proposition 2.3 we can see that A † A ( T − )( T − ) † = P A ∗ ( T − )( T ) = P A ∗ which has closed range. Therefore, the expression of the above theorem in infinite dimensional Hilbertspaces is the following:
Theorem 3.3
Let T ∈ B ( H ) be positive definite and the equation Ax = b ,where A is singular.If the set S = { x : Ax = b } is not empty, the the problem : minimize Φ( x ) = h x, T x i + h p, x i + a, x ∈ S with p, a ∈ H arbitrary has the unique solution x = T − ( AT − ) † ( 12 AT − p + b ) − T − p roof . : Since the existence of a bounded generalized inverse for the operator AT − is proved is Lemma 3.2, the rest of the proof of the above theorem issimilar to the finite dimensional case presented in [8] and is omitted. An interesting case to examine, is when the positive operator T is singular,that is, T is positive semidefinite. In this case, since N ( T ) = ∅ , we have that h x, T x i = 0 for all x ∈ N ( T ) and so, when N ( T ) ∩ S = ∅ this problem will takethe form minimize Φ( x ) = h p, x i + a, f or x ∈ S ∩ N ( T )which is usually treated using the simplex method, when the space is finitedimensional.A different approach in both the finite and infinite dimensional case would be tolook among the vectors x ∈ N ( T ) ⊥ = R ( T ∗ ) = R ( T ) for a minimizing vectorfor Φ( x ). In other words, we will look for the minimum of Φ( x ) under theconstraints Ax = b, x ∈ R ( T ) . Using the fact that T is an EP operator, we will make use of the followingproposition that can be found in Drivaliaris et al [4]: Proposition 3.4
Let T ∈ B ( H ) with closed range. Then the following areequivalent:i) T is EP.ii) There exist Hilbert spaces K and L , U ∈ B ( K ⊕ L , H ) unitary and A ∈B ( K ) isomorphism such that T = U ( A ⊕ U ∗ . We present a sketch of the proof for (1) ⇒ (2): Proof . : Let K = R ( T ), L = N ( T ), U : K ⊕ L → H with U ( x , x ) = x + x , for all x ∈ R ( T ) and x ∈ N ( T ), and A = T | R ( T ) : R ( T ) → R ( T ) . Since T isEP, R ( T ) ⊕ ⊥ N ( T ) = H and thus U is unitary. Moreover it is easy to see that7 ∗ x = ( P T x, P N ( T ) x ) , for all x ∈ H . It is obvious that A is an isomorphism.A simple calculation shows that T = U ( A ⊕ U ∗ . It is easy to see that when T = U ( A ⊕ U ∗ and T is positive, so is A , since h x, T x i = h x , A x i , x ∈ R ( T ).In what follows, T will denote a singular positive operator with a canonical form T = U ( A ⊕ U ∗ , R is the unique solution of the equation R = A and X † = U ( A − )
00 0 U ∗ = U R −
00 0 U ∗ Theorem 3.5
Let T be positive semidefinite and X = T . Consider the equa-tion Ax = b .If the set S = { x : Ax = b } is not empty, then the problem :minimize Φ( x ) = h x, T x i + h p, x i + a, x ∈ S ∩ N ( T ) ⊥ and p, a arbitrary has the unique solution ˆ x = X † ( AX † ) † ( 12 AT † p + b ) − T † p assuming that the operator P A ∗ P T has closed range.Proof . : Since we will restrict the minimization for all vectors x ∈ N ( T ) ⊥ wehave that h x, p i = h x, p i , where p = P R ( T ) p for all vectors p ∈ H , accordingto the decomposition p = p + p ∈ R ( T ) ⊕ N ( T ) . Let x, a, p ∈ H with a and p arbitrary and not a function of T . Hence, thevector ˆ x that minimizes Φ( x ) also minimizesΨ( x ) = h T x, x i + h x, p i + 14 h T † p, p i = h T x, x i + h x, p i + 14 h T † p , p i We can easily see that k Xx + 12 X † p k = Ψ( x )8nd that X = U ( R ⊕ U ∗ , X † = U ( R − ⊕ U ∗ Set y = Xx + X † p , which implies that y ∈ R ( X ) = R ( T ).We have that Xx = y − X † p ⇔ U ( R ⊕ U ∗ x = y − U ( R − ⊕ U ∗ p Hence, x = U ( R − ⊕ U ∗ y − U ( R − ⊕ R − ⊕ U ∗ p and so, since y, p ∈ R ( T ) x = X † y − T † p , with x ∈ R ( T † ) = R ( T ) = N ( T ) ⊥ Since Ax = b , we have that AX † y = b + AT † p and therefore, the minimalnorm solution is ˆ y = ( AX † ) † ( b + 12 AT † p )By substitution, we have thatˆ x = X † ( AX † ) † ( 12 AT † p + b ) − T † p and since T † p = T † p for all p ∈ H ⇒ ˆ x = X † ( AX † ) † ( AT † p + b ) − T † p The only thing that needs to be proved is the fact that the operator AX † has closed range and so its Moore-Penrose Inverse is bounded. Since the twooperators A and X † are arbitrary, one does not expect that the range of theirproduct will always be closed. From Proposition 2.3, this is equivalent to thefact that the operator P A ∗ P T has closed range because A † AX † ( X † ) † = A † AX † X = A † AU ( R − ⊕ R ⊕ U ∗ = P A ∗ P T and the proof is completed. In the sequel, we present an example which clarifiesTheorem 3.5. In addition, the difference between the proposed minimization( x ∈ N ( T ) ⊥ ) and the minimization for all x ∈ H is clearly indicated.9 xample 3.6 Let H = R , and the positive semidefinite matrix Q =
26 10 −
210 8 2 − We are looking for the minimum of the functional f ( u ) = h u, Qu i + h p, u i + a, u ∈ N ( Q ) ⊥ ∩ S with p = (1 , , T , a = (0 , , T and the set of constraints S is defined as S = { ( x, y, z ) : 3 x + y + z = − } . The set N ( Q ) ⊥ has the form u = (2 x − y, x, y ) T , x, y ∈ R .With easy computations, we can see that all vectors u ∈ N ( Q ) ⊥ satisfying theconstraint Au = b , where A = h i and b = − , have the form u = ( x, − x − , − x −
25 ) T The matrices
U, X † are U = √
10 1 √
35 1 √ √ − √ − √
10 3 √
35 3 √ A −
00 0 = . − . − .
063 0 . X † = U ( A − )
00 0 U ∗ = . − . − . − . . . − . . . Using theorem 3.5 we can see that the minimizing vector of f ( u ) under { Au = b, u ∈ N ( Q ) ⊥ } is ˆ u = X † ( AX † ) † ( 12 AQ † p + b ) − Q † p = ( − . , − . , − . T he minimum value of f ( u ) is then equal to 1.4175In Figure 1 we can clearly see that the minimization of the functional f ( u ) forall vectors u ∈ N ( Q ) ⊥ having the form u = (2 x − y, x, y ) , x, y ∈ R , belongingto the plane x + y + z = − , is attained.In Figure 2 we can see that among all vectors belonging to N ( Q ) ⊥ satisfy-ing Au = b , having the form u = ( x, − x − , − x − ) T , x ∈ R , the vector ˆ u =( − . , − . , − . T found from Theorem 3.5 minimizes the functional f ( u ) . −1 −0.5 0 0.5 1−1−0.8−0.6−0.4−0.200.20.40.60.81−1000100200300400500600700800 y− coordinate x− coordinate f ( u ) Au = b f(u)
Figure 1: Constrained minimization of f(u), u ∈ N ( Q ) ⊥ under Au = b We will also examine the minimization of f ( u ) in the cases when u ∈ N ( Q ) andwhen u ∈ R , given that u ∈ S , so that the difference between them is clearlyindicated: ( i ) When the minimization takes place for all vectors u ∈ N ( Q ) , then this f ( u ) Figure 2: f(u), u ∈ S ∩ N ( Q ) ⊥ problem takes the form : minimize f ( u ) = h p, u i , u ∈ S .We can see with easy calculations, that the only vector u ∈ N ( Q ) ∩ S is ˜ u = (0 . , − . , . T . In this case , f (˜ u ) = h p, ˜ u i = 1 . ii ) When the minimization takes place for all vectors u ∈ R , u ∈ S , thenthe minimizing vector of f ( u ) is w = ( − . , . , − . T and theminimum value of f ( u ) is f ( w ) = − . . Corollary 3.7
We can easily see that, in the case when the vectors p, a areboth equal to zero, the functional is a constrained quadratic form Φ( x ) = h T x, x i under Ax = b .In this case, the minimizing vector ˆ u belonging to N ( T ) ⊥ is then equal to ˆ u = X † ( AX † ) † b as it was discussed and proved in [10]. In this work, we propose a constrained minimization in the case of a quadraticfunctional related to a positive semidefinite operator. The proposed minimiza-12ion takes place for all vectors perpendicular to the kernel of the correspondingoperator. This proposed constrained minimization method has the advantage ofa unique solution and is easy to implement. Practical importance of this resultin the finite dimensional case, can be in numerous applications such as networkanalysis, filter design, spectral analysis, direction finding etc. In many of thesecases, the knowledge of the non zero part of the solution corresponding to thematrix may be of importance.