Low rank solutions to differentiable systems over matrices and applications
aa r X i v : . [ m a t h . O C ] M a y Low rank solutions to differentiablesystems over matrices andapplications ∗ Thanh Hieu LEDepartment of Mathematics, Quy Nhon University, Vietnam
Abstract
Differentiable systems in this paper means systems of equationsthat are described by differentiable real functions in real matrix vari-ables. This paper proposes algorithms for finding minimal rank solu-tions to such systems over (arbitrary and/or several structured) ma-trices by using the Levenberg-Marquardt method (LM-method) forsolving least squares problems. We then apply these algorithms tosolve several engineering problems such as the low-rank matrix com-pletion problem and the low-dimensional Euclidean embedding one.Some numerical experiments illustrate the validity of the approach.On the other hand, we provide some further properties of low ranksolutions to systems linear matrix equations. This is useful when thedifferentiable function is linear or quadratic.
Keywords: rank minimization problem, generalized Levenberg-Marquardtmethod, positive semidefinite matrix, low-rank matrix completion problem,Euclidean distance matrix ∗ The research is supported by Vietnam National Foundation for Science and TechnologyDevelopment (NAFOSTED) under grant number 101.01-2014.30.Email: [email protected] Motivation and preliminaries
Several problems in either engineering or computational mathematics can bereformulated as rank minimization problems (shortly, RM-problems) in theform minimize rank ( X )subject to X ∈ C , (1)where C is a subset of R m × n , the set of all m by n matrices with real entries.RM-problem (1) is computationally NP-hard in general, even when C is anaffine subset of R m × n . There hence is a number of algorithms for solving thisproblem with respect to special cases of C , see, eg., [9,16,24] and the referencesthere in. When the constraints are defined by linear matrix equations, i.e., C is the solution set of a linear system of equations ℓ ( X ) = b ∈ R k , the presentproblem is called affine rank minimization problem (shortly, ARM-problem)and is in the form [24] minimize rank ( X )subject to X ∈ R m × n ,ℓ ( X ) = b. (2)When the constraint region is considered on the cone of positive semidefinitematrices, the authors in [20] relaxed the non-convex rank objective functionin problem (2) into the nuclear norm that is a convex function. The wholeproblem is then a semidefinite program [30] and can be efficiently solved bySDP solvers. In our point of view, by using the Cholesky decomposition,each positive semidefinite matrix X can be written as X = Y Y T , Y ∈ R n × n . The linear map in the later problem (2) now becomes a quadratic map in Y. In this paper, we focus on the problem over a more general set C , incomparison with the sets we have discussed above. Such a set is determinedby a differentiable map. That is, we focus on the problemminimize rank ( X )subject to X ∈ C ,φ ( X ) = b, (3)where C ⊆ R m × n and φ : R m × n → R k is a differentiable map. This func-tion is clearly non-convex in general. Our method applies the generalized2evenberg-Marquardt method [27] for checking whether there exists a so-lution of rank r, step by step, for r = 1 , , . . . The differentiability of φ guarantees for the existence of its Jacobian in the Levenberg-Marquardtsteps. It turns out that the problem of finding a matrix of rank r =1 , , . . . , min { m, n } , solving the equation φ ( X ) = b is the most importantin our method.We now recall some important results on matrix factorization in linearalgebra that are used in the paper.By . T we denote the transpose of matrices. For a real symmetric matrix A, i.e., A T = A, by A (cid:23) A is positive semidefinite, i.e., x T Ax ≥ x ∈ R n . This, equivalently, means its eigenvalues are all non-negative.For any two real symmetric matrices A and B, we write A (cid:23) B if A − B (cid:23) . Let S n denote the set of n by n real symmetric matrices, and S n + denote thecone of positive semidefinite matrices in S n . Proposition 1. (see, e.g, [4] or [12, Section 2.6, Observation 7.1.6])
Anypositive semidefinite matrix (PSD matrix) A ∈ S n + has a Cholesky decom-position A = LL T , where L ∈ R n × n is a lower triangular matrix which iscalled a Cholesky factor of A. In particular, if r = rank ( A ) then one can find L ∈ R n × r . Another fact is that for two matrices
A, B ∈ S n , then A (cid:23) B if and onlyif P T AP (cid:23) P T BP for any nonsingular matrix P ∈ R n × n . Proposition 2. (see e.g, [12, Section 0.4.6]
Let A be an m × n real matrix.Then i) rank ( A ) = rank ( A T ) = rank ( AA T ) = rank ( A T A ) . ii) A ∈ R m × n has rank r if and only if there exist matrices X ∈ R r × m ,Y ∈ R r × n and B ∈ R r × r nonsingular with rank ( X ) = rank ( Y ) = r suchthat A = X T BY.
A consequence, A can be written as A = X T Z with Z = BY ∈ R r × n and rank ( X ) = rank ( Z ) = r. Proposition 3.
For any linear map ℓ : R m × n → R one can find a matrix A ∈ R m × n such that ℓ ( X ) = Tr ( A T X ) = Tr ( AX T ) , ∀ X ∈ R m × n . Specially, if ℓ : S n → R then A can be found in S n , i.e., A T = A. roof. Suppose ℓ : R m × n → R is a linear map. Consider R m × n as a realvector space endowed with the basis { E ij | i = 1 , . . . , m ; j = 1 , . . . , n } , where E ij is the m × n matrix whose entries are zeros except for the ( i, j )th onebeing 1. Let A = [ ℓ ( E ij )] j =1 ,...,ni =1 ,...,m ∈ R m × n . Then for every X = [ x ij ] ∈ R m × n ,X = P i,j x ij E ij , we have ℓ ( X ) = m X i =1 n X j =1 x ij ℓ ( E ij ) = Tr ( AX T ) = Tr ( A T X ) . If ℓ : S n → R then it follows that ℓ ( X ) = m X i =1 n X j =1 x ij ℓ ( E ij ) = n X i =1 x ii ℓ ( E ii ) + X ≤ i Jacobian matrix ) of f is the m × n matrix defined by Jac f ( x ) , ∂f ( x ) ∂x = ∂f ( x ) ∂x . . . ∂f ( x ) ∂x n ... . . . ... ∂f m ( x ) ∂x . . . ∂f m ( x ) ∂x n ∈ R m × n . We now recall a general definition for the derivative of a matrix valuedfunction . Suppose F : R m × n → R p × q is a ( p × q )-matrix valued function ofan ( m × n )-matrix variable X. Suppose that F = [ F rs ] ∈ R p × q and we definethe derivative of this function as the pq × mn matrix Jac F ( X ) , ∂ vec F ( X ) ∂ vec X = ∂F ( X ) ∂x ∂F ( X ) ∂x . . . ∂F ( X ) ∂x mn ∂F ( X ) ∂x ∂F ( X ) ∂x . . . ∂F ( X ) ∂x mn ... ... . . . ... ∂F pq ( X ) ∂x ∂F pq ( X ) ∂x . . . ∂F pq ( X ) ∂x mn ∈ R pq × mn , where vec X ∈ R mn × denotes the vector obtained by stacking its columnsone underneath the other, i.e., if X ∈ R m × n and X j , j = 1 , . . . , n, are thecolumns of X then vec X = (cid:2) X T1 . . . X T n (cid:3) T . 4e list below the important properties of the derivative of trace functionsthat will be used in either paper or Matlab codes (see, eg., [22]). • Let A be a given matrix in R m × n . If F : R m × n → R is defined by F ( X ) = Tr ( A T X ) , ∀ X ∈ R m × n , then Jac F ( X ) = vec ( A ) T . (4) • Let A ∈ R n × n and B ∈ R m × m be given. If F : R m × n → R is defined by F ( X ) = Tr ( XAX T B ) then Jac F ( X ) = vec ( B T XA T + BXA ) T . (5) • Let A, B be two given matrices in R m × m . Then for all X ∈ R m × m , JacTr ( AXB ) = vec ( A T B T ) T and JacTr ( AX T B ) = vec ( BA ) T . (6)This paper is organized as follows. Section 2 presents the main algorithmfor solving problem (3). This algorithm will be applied to particular problemswith respect to several types of constraint sets. The affine rank minimiza-tion problem over arbitrary matrices is presented in Section 3 and a similarmethod applied for positive semidefinite matrices is handled in Section 4.Section 5 summarizes some applications of our solution method to severalproblems in engineering. The corresponding numerical experiments are ex-hibited in Section 6. The last section presents the conclusion and discussionfor the future works. In this work, with the help of Proposition 2, we solve problem (3) by usingthe generalized Levenberg-Marquardt method [27] to find a matrix X ∈ C ⊂ R m × n step by step for rank ( X ) = 1 , , . . . , min { m, n } such that φ ( X ) = b. In this situation, we consider the least square problem with respect to thefunction F : R µ → R k , with appropriate integer number µ, whose coordinatefunctions are defined by F j ( X ) = φ j ( X ) − b j , ∀ j = 1 , . . . , k, ∀ X ∈ C . (7)We can summary this algorithm as follows.5 lgorithm 1. Find minimal-rank matrix solving problem (3). Input : Scalars b , . . . , b k and function φ. Output : a solution X ∈ C ⊂ R m × n to (3) .1. Set r = 1 . 2. Solve system (7) by applying the Levenberg-Marquardt method [27].3. If (7) has a numerical solution then stop.Else, set r = r + 1 and go to Step 2.In fact, to perform the experiments, the variable matrices are vectorized.Namely, the functions F j in (7) is vec ( X ) . This suggests us to study the rankone solutions to systems of equations. In this section we are concentrating on numerically solving ARM-problem (2).Using Proposition 3 each of the k linear equations ℓ i ( X ) = b i , i = 1 , , . . . , k is written as ℓ i ( X ) = Tr ( A T i X ) = Tr ( A i X T ) = b i . The function of the least square problem in this case is determined as: F j ( X ) = ℓ j ( X ) − b j , j = 1 , . . . , k. It is clear that such a matrix X of rank r can be found only first r columns X (: , r ) and its n − r last ones are identified to zero vectors. However, wewill see later this may not applicable in some particular cases, for example,the matrix completion problem below. A modification is to apply Proposition2 to find X as X = Y T Z for two matrix variables Y ∈ R r × m and Z ∈ R r × n . Namely we now focus on the following problemminimize rank ( Y T Z )subject to ( Y, Z ) ∈ R r × m × R r × n ,ℓ ( Y T Z ) = b. (8)6roblem (8) is then a special case of problem (3) with φ ( Y, Z ) = ℓ ( Y T Z ) . To perform this modification, we need the following auxiliary results. Set W = [ Y Z ] ∈ R r × ( m + n ) for each ( Y, Z ) ∈ R r × m × R r × n . Recall that thelinear map ℓ : R m × n → R k is defined by k matrices A , . . . , A k ∈ R m × n . Thatis ℓ ( U ) = (cid:2) Tr ( A T1 U ) . . . Tr ( A T k U ) (cid:3) T , ∀ U ∈ R m × n . For each r = 1 , , . . . , p = min { m, n } , the least squares problem in thissituation is then defined by the function F : R r × ( m + n ) ≡ R r × m × R r × n −→ R k ,F ( W ) = F ( Y, Z ) = ℓ ( Y T Z ) − b, ∀ W = ( Y, Z ) ∈ R r × m × R r × n . It is clear that each coordinate function F i : R r × ( m + n ) ≡ R r × m × R r × n −→ R is determined by F i ( W ) = F i ( Y, Z ) = Tr ( A T i Y T Z ) − b i , ∀ W = ( Y, Z ) ∈ R r × m × R r × n . The Jacobian matrix of F can hence be calculated as Jac ( F ) = ∂F∂W = ∂ vec F∂ vec W = ∂F ∂W ... ∂F k ∂W = ∂F ∂Y ∂F ∂Z ... ... ∂F k ∂Y ∂F k ∂Z ∈ R k × r ( m + n ) , where vec ( W ) = [ vec ( Y ) T vec ( Z ) T ] T ,∂F i ∂Y = ∂ Tr ( A T i Y T Z ) ∂Y = vec ( ZA T i ) T ,∂F i ∂Z = ∂ Tr ( A T i Y T Z ) ∂Z = vec ( Y A i ) T . Algorithm 1 will find a solution W = [ Y Z ] and then a solution to problem(8) can be defined as X = Y T Z. Some corresponding numerical results arepresented in Section 6. In this section we focus on the ARM-problem for semidefinite matrices (3).By Proposition 3, we can characterize the linear map ℓ by k symmetric ma-trices A , . . . , A k ∈ S n , with respect to b , . . . , b k . In the subsection below, we7evelop some more properties of the solutions to a system of linear equations.This might be not for our algorithm but it could be useful information inliterature. Consider the system of linear equations as follows: Tr ( A i X ) = b i , i = 1 , . . . , m, (9)where A i , X are real symmetric of order n and b = [ b . . . b m ] T ∈ R m . Thecorresponding homogeneous of system (9) is defined by Tr ( A i X ) = 0 , i = 1 , . . . , m. (10)On the other hand, for such a nonhomogeneous system (9), we call the system Tr ( ˜ A i ˜ X ) = 0 , i = 1 , . . . , m, (11)with ˜ A i = (cid:20) A i − b i (cid:21) , its “ dominating system ”.We also note that system (9) can be written in the classical form: svec ( A i ) T svec ( X ) = b i , or A svec ( X ) = b, (12)where A = svec ( A ) T ... svec ( A m ) T ∈ R m × τ ( n ) . Such a system has τ ( n ) variables. It is well known by Kronecker-Capellitheorem [15] that the system A svec ( X ) = b has a solution if and only if rank ( A ) = rank ( ˜ A ) , where ˜ A := [ A | b ] . Moreover, if rank ( A ) = rank ( ˜ A ) = r then such a system has only one solution when r = τ ( n ) . In the case r < τ ( n ) , such a system has many solutions in which r variables linearlydependent on τ ( n ) − r other variables. We also note that the system { A i } is linearly (in)dependent in S n if and only if so is the system { svec ( A i ) } in R τ ( n ) . The following result gives us an equivalence of the non-homogeneous lin-ear system and a homogeneous linear system in one more variable, in thecase that the matrices A i are linearly independent.8 roposition 4. System (9), with linearly independent matrices A i ’s and b = 0 , m ≤ τ ( n ) , has a solution (must be nonzero) if and only if system (11)has a nontrivial solution.Moreover, if the positive semidefiniteness of a solution to one of these twosystems is valuable then so is a solution to the other system.Proof. If 0 = X = [ x ij ] ∈ S n is a solution to (9) then one can check0 = ˜ X = (cid:20) X n × × n (cid:21) ∈ S n +1 is a solution to (11) since Tr ( ˜ A i ˜ X ) = Tr ( A i X ) − b i = 0 , ∀ i = 1 , . . . , m. For the opposite direction, we first note that if the homogeneous domi-nating system (11) has nonzero solutions then there exists one whose ( n +1 , n + 1)st entry is nonzero. Indeed, since { A i } mi =1 is linearly independent, sois { ˜ A i } mi =1 . But the homogeneous dominating system (11) has m equationsand τ ( n + 1) variables. Its solution vector space is hence of τ ( n + 1) − m > t := ˜ x ( n +1)( n +1) = 0 . Indeed, if every solution t was zeros then there would exist a nonsingularmatrix P (exists from the Gaussian elimination) such that P ˜ A = • • . . . • • . . . • ... ... . . . ...0 0 · · · . This implies rank ( A ) < m. This contradicts to the fact that { A i } mi =1 islinearly independent.With a solution ˜ X satisfying the above discussion, let X be the n × n leading principle submatrix of ˜ X, we have0 = Tr ( ˜ A i ˜ X ) = Tr ( A i X ) − b i t, i = 1 , . . . , m. This t X is a solution of (9).The rest of the proposition is an immediate consequence of what haveshown above. 9 emark . Even though some nonzero solutions of two systems (9) and (11)stated in Proposition 4 simultaneously exist, they do not need have the samerank. To see this, let us consider the linear system Tr ( A X ) = Tr ( A X ) = 0 , Tr ( A X ) = − , where A = diag (1 , − , , A = diag (1 , , − 1) and A = and they are linearly independent. The matrices ˜ A i are then defined as˜ A = diag (1 , − , , , ˜ A = diag (1 , , − , − 1) and ˜ A = . It is shown in [32] that the dominating homogeneous system (11) has no rank-one solution but a rank-three solution ˜ X = diag (1 , , , . In our situation,we can find a rank-two solution, for example, ˜ Y = " − . However,the initial non-homogeneous system defined by the matrices A , A , A has arank-one solution X = (cid:20) (cid:21) . The following proposition tells us the relationship between the existenceof a positive definite element in Span ( A , . . . , A m ) := { m P i =1 t i A i : t i ∈ R } andthat of trivial solution of the system Tr ( A i X ) = 0 , i = 1 , m over S n + . This isdue to Bohnenblust [5] and is restated in some equivalent versions in [1,13,32].In the their works, the proofs are mainly based on either the separationtheorem for two nonempty convex sets (see, eg., [2, Theorem III.1.2]) or theSDP duality theory (see, eg., [31]). In our situation, we use only knowledgeon linear algebra, in particular, the theory of orthogonal complement in aninner-product vector space. This also gives us a stronger result, comparedwith the existence one. Proposition 5. [13, 32] With the notation above, we have { X ∈ S n + | Tr ( A i X ) = 0 , ∀ i = 1 , m } = { } ⇐⇒ S n ++ ∩ Span ( A , . . . , A m ) = ∅ . 10e have already known by Proposition 1 i) that any positive semideifnitematrix X ∈ S n + with rank ( X ) = r can be expressed as X = r P i =1 x i x Ti for some x i ∈ R n , i = 1 , . . . , r. Based on the fact Tr ( A Ti X ) = Tr ( A i r X i =1 x j x Tj ) = r X j =1 x Tj A i x j , ∀ i = 1 , . . . , m, the problem of finding a low-rank solution to (9) is of the formˆ x T b A i ˆ x = b i , i = 1 , . . . , m, (13)where the coefficient matrices now are b A i := ⊕ rj =1 A j and ˆ x = [ x T . . . x Tr ] T . It is clear that a nonzero solution to (13) gives a solution to system (9) withrank less than or equal to r. This is because of that x , . . . , x r might belinearly dependent. We thus have the following. Proposition 6. If system (9) has a solution of rank r then system (13) hasa nonzero solution. Conversely, if system(13) has a nonzero solution thensystem (9) has a solution of rank less than or equal to r. Remark . i) When system (9), with b = 0 , has a nonzero positive semidef-inite solution then by the work of Barvinok [3], there is another positivesolution with the rank at most √ m + 1 − . This upper bound is smallerthan n since m ≤ τ ( n ) . For us this bound is sharpest by now.For homogeneous system (10), this bound does not necessary hold [32].ii) According to the works [7, 18], one obtainsmax { rank X : X ∈ Span ( A , . . . , A m ) } ≥ n + 1 − p (2 n + 1) − m . This, indeed, follows from the proofs for lower bound in [7, 18]. Even though this is a particular case of the ARM-problem over arbitrarymatrices, Proposition 1 allows us to find a Cholesky factor Y ∈ R n × r insteadof a positive semidefinite matrix, and this leads to a reduction in number of11ariables for the ARM-problem. So problem (3) can be cast in the followingformminimize rank ( Y )subject to [ Tr ( A T1 Y Y T ) . . . Tr ( A T k Y Y T )] T = ℓ ( Y Y T ) = b. (14)The idea for solving this problem is similar to the previous case, where onechecks whether there exists a matrix with lowest possible rank satisfying therequirements. In this situation, at the step corresponding to r, the followingfunction is applied: F : R n × r −→ R k defined by F ( Y ) = ℓ ( Y Y T ) − b, ∀ Y ∈ R n × r . The coordinate functions F i : R n × r −→ R k are obviously defined by F i ( Y ) = Tr ( A T i Y Y T ) − b i , ∀ Y ∈ R n × r . The Jacobian matrix of F in this case follows from (5): Jac ( F ) = ∂F∂Y = ∂ vec F∂ vec Y = ∂F ∂Y ... ∂F k ∂Y ∈ R k × rn , where ∂F i ∂Y = ∂ Tr ( A T i Y Y T ) ∂Y = vec [( A T i + A i ) Y ] T . In this section, we consider three applications of problem (3). In machine learning scenarios, e.g., in factor analysis, collaborative filtering,and latent semantic analysis [24,25,28], there are several problems that can bereformulated as the low-rank matrix completion problem. Given the values ofsome entries of a matrix, this problem fills the missing entries of the matrix12uch that its rank is small as possible. This problem is summarized andreformulated as follows. Given a set of triples( R, C, S ) ∈ { , . . . , m } k × { , . . . , n } k × R k , and we wish to construct a small-as-possible rank matrix X = [ X rs ] ∈ R m × n , such that X R ( i ) ,C ( i ) = S ( i ) for all i = 1 , . . . , p. This can be reformulated asminimize rank ( X )subject to X R ( i ) ,C ( i ) = S ( i ) , ∀ i = 1 , . . . , p. (15)This problem can then be solved by using Algorithm 1. Euclidean distance matrices, shortly EDMs, have received increased attentionbecause of its many applications which can be found in eg., [6, 8, 21, 24] andreferences there in.We first recall this problem. Let D = [ d ij ] ∈ S n be a Euclidean distancematrix (EDM) associated to the points x , . . . , x n ∈ R r , i.e., d ij = k x i − x j k = x T i x i + x T j x j − x T i x j , i, j = 1 , . . . , n. (16)The smallest positive integer number r is said to be the embedding dimension of D. Following [24], let ∈ R n × be the column vector of ones. Define V , I n − n T . Note that V is the orthogonal projection matrix onto the hyperplane { v ∈ R n × : T v = 0 } . In particular, rank ( V ) = n − V has an eigenvector with respect to the eigenvalue zero. It followsfrom the work in [26] that D is an EDM of n points in R r if and only if threefollowing conditions hold: d ii = 0 , ∀ i = 1 , . . . , n ; − V DV (cid:23) rank ( V DV ) ≤ r. n and partial Euclidean matrix D , i.e.,every entry of D is either “specified” or “unspecified”, diag ( D ) = 0 , andevery fully specified principal sub-matrix of D is also a Euclidean distancematrix. The low-dimensional Euclidean embedding problem finds a Euclideanmatrix D consistent with the known pairwise distances described by D andassociated to a number of points in the smallest dimensional space R r . Sucha problem can be reformulated as the ARM-problem [24]minimize rank ( V DV )subject to − V DV (cid:23) ,ℓ ( D ) = b, (17)where ℓ : S n → R p is an appropriate linear map, corresponding to the spec-ified entries in D , including the condition that makes the diagonal of D tobe zero. If one sets X = [ x . . . x n ] ∈ R r × n then D can be found in form D = D ( X ) := diag ( X T X ) T + diag ( X T X ) T − X T X (18)because of (16). Since V = 0 , − V D ( X ) V = 2 V X T XV. (19)Substituting this fact into problem (17) we get the equivalent one:minimize rank ( XV )subject to X ∈ R r × n ,ℓ ( D ( X )) = b. (20)It is clear that the above problem is of the form of problem (3) with φ ( X ) = ℓ ( D ( X )) . We make the linear map ℓ more explicit as in [14]. Let H be the 1-0adjacency matrix, i. e., h ij = (cid:26) i, j ) ∈ E, i, j ) E, for the set of subscripts E corresponding to the specified entries of D . Themain problem is to find an as-small-as-possible rank completion D of D . D in the form D = diag ( Z ) T + diag ( Z ) T − Z,Z = X T X,H ⊙ D = H ⊙ D , (21)where ⊙ denotes the component-wise (or Hadamard) matrix product. Withthe help of the fact rank ( XV ) ≤ rank ( X ) , problem (21) is then reduced tothe rank minimization problem in the formminimize rank ( X )subject to X ∈ R r × n ,H ⊙ D ( X ) = H ⊙ D . (22)In many applications one seeks the embedding dimension being two or three.Note that when D is determined as D = D ( X ) , the following formulais used to compute the Jacobian matrices used in the Levenberg-Marquardtalgorithm: ∂d ij ∂X = " . . . | {z } r times . . . . . . | {z } r times x T i − x T j ) 0 . . . x T j − x T i ) 0 . . . ∈ R × nr . Here we assume i ≤ j. From the theoretical point of view, the rank minimization problem is NP-hardso that there has not been any method directly solve this one in the literature.A good way for solving the RM-problem over positive semidefinite matricesis to solve the corresponding problem that minimize the nuclear norm (see,eg., [19, 24]). The nuclear norm minimization problem (NNM-problem) isa really good one to give suitable lower and upper bounds for the origi-nal RM-problem. Additionally, in [24] it is proved that the NNM-problemover general matrices is tractable to solve since it can be reformulated as asemidefinite program [10]. Another method for solving the NNM-problem15ver positive semidefinite matrices was proposed in [19] by using ModifiedFixed Point Continuation Method.We now illustrate the RM-problem over generic matrices. Table 1 showsthe results obtained by Algorithms 1 for this case. The matrices A , . . . , A k are randomly chosen with entries in (0 , . The backward errors are deter-mined by err = k ℓ ( X ) − b k k b k . The result for each case is averagely taken per three experiments. The numer-ical results show that Algorithm 1 gives better solutions if the factorizationin Proposition 2 is applied. More precisely, we see in Table 1, the resultswhen Proposition 2 is applied have smaller rank. Table 1 also exhibits acomparison between our method and the one described in [24].The NNM-problem approximating problem (2) is followed in [24] and canbe summarize as follows minimize k X k ∗ subject to ℓ ( X ) = b, (23)where k . k ∗ denotes the nuclear norm of X, which is the sum of all its singularvalues. If X has a singular value decomposition X = U Σ V T then one cansolve problem (23) by solving the semidefinite program:minimize ( Tr ( W ) + Tr ( W ))subject to (cid:20) W XX T W (cid:21) (cid:23) ,ℓ ( X ) = b. (24)This is nice formulation in theoretical point of view but in practice the result-ing matrices may have “high-rank” by SDP solvers . One can see in Table 1,where problem (24) is implemented in CVX toolbox [11] of Matlab calling Sedumi [29], that the semidefinite program seems to give resulting matriceswith full rank and less accuracy.For the RM-problem over positive semidefinite matrices, the experimentsperform with randomly chosen symmetric matrices A , . . . , A k . The resultfor each case is also averagely taken per three experiments.16 n k X (: , r ) rank ( X ) err X = Y T Z rank ( X ) err SDP rank ( X ) err5 6 4 1 5.31e-16 1 2.44e-16 5 1.02e-0951 50 51 1 7.13e-15 1 4.49e-15 50 8.42e-1050 100 81 2 5.56e-16 1 9.42e-16 50 3.38e-0950 200 100 3 5.46e-16 1 7.31e-16 50 4.08e-09100 200 300 3 2.63e-14 2 3.83e-15 out of memory500 550 300 1 1.66e-15 1 1.93e-14 out of memory500 500 450 1 2.04e-15 1 1.10e-14 out of memoryTable 1: Comparison between LM-method and SDP solving the RM-problemover m × n matrices.Table 2 shows a comparison between our method and the one describedin [19]. The errors in this table are computed aserr = k ℓ ( X ) − b k k b k . What we see in Table 2 that the values of the rank of resulting matrices n k rank (X)LM AFPC-BB errLM AFPC-BB100 579 6 10 1.89e-16 9.46e-4200 1221 7 10 1.87e-15 9.84e-4500 5124 11 10 2.52e-15 4.90e-3500 3309 7 27 3.00e-15 NATable 2: Comparison between LM-method and AFPC-BB solving the RM-problem over positive semidefinite matrices. The error of the method AFPCfor the case ( n, k ) = (500 , m, n. We take R, C ∈ N k with the entries are random in { , . . . , m } , { , . . . , n } , respectively, and so is17 ∈ (0 , k . For the cases m = n, it turns out the results for the systems ofquadratic equations. More precisely, the system has solution if the solutions’have rank one. m n k rank (X) m n k rank (X)5 6 4 4 50 50 51 251 50 51 3 100 100 50 150 100 81 1 150 150 100 150 200 100 1 200 200 200 2100 200 300 3 400 400 350 1500 550 300 1 500 500 450 1Table 3: Solution to the low-rank matrix completion using LM-method. This section shows the numerical results for problem (22). All tests aredealt with partial Euclidean matrices D with entries randomly taken in theinterval [0 , . Table 4 shows the Euclidean embedding dimensions for all cases that D are dense, i.e., the entries of the corresponding matrix H are all one. n rank (X) err n rank (X) err4 2 5.09e-16 100 2 1.33e -1410 2 9.38e-16 150 2 1.99e-1420 2 2.09e-15 200 2 2.54e-1430 2 4.04e-15 300 2 3.84e-1440 2 6.10e-15 400 2 4.20e-1450 2 5.98e -15 500 2 4.32e-14Table 4: Solution to EDM problems with respect to dense partial EDMmatrices.Table 5 shows the results for sparse matrices D , i.e., the entries of thecorresponding matrix H are either zero or one.18 rank (X) err n rank (X) err4 2 1.95e-16 100 2 4.03e -1610 2 5.16e-16 150 2 3.97e-1620 2 4.56e-16 200 2 4.23e-1630 2 4.85e-16 300 2 3.89e-1640 2 4.15e-16 400 2 4.42e -1650 2 4.00e -16 500 2 4.54e -16Table 5: Solutions to EDM problems with respect to randomly-chosen sparsepartial EDM matrices.The backward errors of all tests in both cases of D are determined aserr = k D − D k k D k . It turns out that the configurations of our experiments are all in two dimen-sional spaces. We have proposed an algorithm for solving the rank minimization problemover a subset of R m × n determined by a differentiable function. As a conse-quence, the affine rank minimization problems over either arbitrary or posi-tive semidefinite matrices have been numerically tested. This algorithm wasthen applied to solve the low-rank matrix completion problem and the low-dimensional Euclidean embedding problem. Some numerical experimentshave been performed to illustrate our algorithms as well as the applications.We have also developed some useful properties for low rank solutionsto systems of linear matrix equations. This suggests us a reformulation ofthe IIR and FIR low-pass filter problems described in [17] as optimizationproblems over rank-one positive semidefinite matrices. In the future we willdeal with this method to solve such filter design problem. This might besuitable because the resulting positive semidefinite matrices derived by SDPsolvers in [17] are usually full rank. Obviously, this requires a much moreamount of memory and complexity in comparison with rank-one setting.19 cknowledgement. The author would like to thank Prof. Marc Van Barelfor his valuable discussion which led to improvements in the manuscript andthe Matlab codes. References [1] A. Barvinok, A Remark on the Rank of Positive Semidefinite MatricesSubject to Affine Constraints , Discrete Comput. Geom. (2001), no.1, 23–31.[2] A. Barvinok. A Course of Convexity , volume 54 of Graduate Studies inMathematics . American Mathematical Society, Providence RI, 2002.[3] A. I. Barvinok. Problems of distance geometry and convex properties ofquadratic maps . Discrete Comput. Geom. (1995), 189–202.[4] R. Bhatia. Positive Definite Matrices . Princeton Series in Applied Math-ematics. Princeton University Press, 2007.[5] F. Bohnenblust. Joint positiveness of matrices . Technical re-port, California Institute of Technology, 1948. Avaliable at http://orion.uwaterloo.ca/ hwolkowi/henry/book/fronthandbk.d/Bohnenblust.pdf .[6] I. Borg and P. J. F. Groenen. Modern Multidimensional Scalling: Theoryand Applications . Springer Science+Business Media, Inc., 2005.[7] M.-D. Choi, T Y Lam, and B Reznick, Sums of squares of real polyno-mials, in Proceedings of Symposia in Pure Mathematics , vol. 58, 1995,pp. 103–126.[8] J. Dattorro, Convex optimization & Euclidean distance geometry , Me-boo, 2005.[9] M. Fazel, Matrix rank minimization with applications , PhD thesis,Stanford University, March 2002.[10] M. Fazel, H. Hindi, and S. Boyd, A Rank Minimization Heuristic withApplication to Minimum Order System Approximation, In Proceedingsof the American Control Conference, Arlington, VA , 2001, pp. 4734–4739. 2011] M. Grant and S. P. Boyd, CVX : Matlab software for disciplined convexprogramming, version 2.1 , June 2015. http://cvxr.com/cvx .[12] R. A. Horn and C. R. Johnson, Matrix analysis , Cambridge UniversityPress, 1985.[13] I. Klep and M. Schweighofer, An Exact Duality Theory for SemidefiniteProgramming Based on Sums of Squares . Math. Oper. Res. (2013),no. 3, 569–590.[14] N. Krislock and H. Wolkowicz, Euclidean Distance Matrices and Ap-plications. In M. F. Anjos and J. B. Lasserre, editors, Handbook onSemidefinite, Conic and Polynomial Optimization , pp. 879–914, Inter-nat. Ser. Oper. Res. Management Sci., 166, Springer, New York, 2012.[15] A. G. Kurosh, Higher Algebra (English translation from Russian by G.Yankovsky) . Moscow Mir Publishers, 1980.[16] T. H. Le, L. Sorber, and M. Van Barel, The Pythagoras number of realsum of squares polynomials and sum of square magnitudes of polynomi-als, Calcolo (2013), no. 4, 283–303.[17] T. H. Le and M. Van Barel, A convex optimization model for findingnon-negative polynomials. J. Comput. Appl. Math. (2016), 121–134.[18] T. H. Le and M. Van Barel, On bounds of the Pythagoras number ofthe sum of square magnitudes of laurent polynomials. Numer. AlgebraControl Optim. (2016), no. 2, 91–102.[19] Y. Ma and L. Zhi, The Minimum-Rank Gram Matrix Completionvia Modified Fixed Point Continuation Method, in: ISSAC 2011–Proceedings of the 36th International Symposium on Symbolic and Alge-braic Computation , 241–248, ACM, New York, 2011.[20] M. Mesbahi and G. P. Papavassilopoulos, On the rank minimizationproblem over a positive semidefinite linear matrix inequality. IEEETrans. Automat. Control (1997), no. 2, 239–243.[21] R. Parhizkar, Euclidean Distance Matrices: Properties, Algorithms andApplications , PhD thesis, ´Ecole polytechnique f´ed´erale de Lausanne,2013. 2122] K. B. Petersen and M. S. Pedersen, The matrixcookbook , Technical University of Denmark, 2012. .[23] A. Prestel and C. N. Delzell. Positive Polynomials , volume 53 of Springermonographs in mathematics . Springer, 2013.[24] B. Recht, M. Fazel, and P. A. Parrilo, Guaranteed minimum-rank solu-tions of linear matrix equations via nuclear norm minimization, SIAMRev. (2010), no. 3, 471–501.[25] J. D. M. Rennie and N. Srebro, Fast maximum margin matrix factor-ization for collaborative prediction, in: Proceedings of the 22nd Inter-national Conference of Machine Learning , 713–719, 2005.[26] I. J. Schoenberg, Remarks to Maurice Fr´echet’s article “Sur la d´efinitionaxiomatique d’une classe d’espaces distanci´es vectoriellement applicablesur l’espace de Hilbert”, Ann. of Math. (1935), no. 3, 724–732.[27] L. Sorber, M. Van Barel, and L. De Lathauwer, Unconstrained opti-mization of real functions in complex variables. SIAM J. Optim. (2012), no. 3, 879–898.[28] N. Srebro, Learning with Matrix Factorizations , Phd thesis, Mas-sachusetts Institute of Technology, Cambridge, MA, 2004.[29] J. F. Sturm. Using SeDuMi 1.02, a Matlab toolbox for optimizationover symmetric cone. Optim. Methods Softw. (1999), no. 1-4,625–653.[30] L. Vandenberghe and S. P. Boyd, Semidefinite programming, SIAMRev. (1996), 49–95.[31] H. Wolkowicz, R. Saigal, and L. Vandenberghe, Handbook of semidefiniteprogramming: theory, algorithms and applications , Kluwer AcademicPublishers, 2000.[32] Y. B. Zhao and M. Fukushima, Rank-one solutions for homogeneouslinear matrix equations over the positive semidefinite cone, Appl. Math.Comput.219