TTensor Transpose and Its Properties
Ran Pan
Department of MathematicsUniversity of California, San DiegoEmail: [email protected]
November 7, 2014
Abstract
Tensor transpose is a higher order generalization of matrix transpose. In this paper,we use permutations and symmetry group to define the tensor transpose. Then we discussthe classification and composition of tensor transposes. Properties of tensor transpose arestudied in relation to tensor multiplication, tensor eigenvalues, tensor decompositions andtensor rank.
Keywords. tensor, transpose, symmetry group, tensor multiplication, eigenvalues, decom-position, tensor rank.
A tensor is a multidimensional or N -way array. The order of a tensor is the number of itsdimensions. If X denotes a real N -order tensor, we have X ∈ R I × I ×···× I N . I × I × · · · × I N iscalled the size of X . It’s clear that tensor X with entries X ( i , i , . . . , i N ) has N indices. There’resome examples: a vector (i.e., a 1-order tensor), a matrix (i.e., a 2-order tensor)and a 3-ordertensor are shown in Figure 1.1.In recent decades, research of tensors attracted much attention. In theoretical work, thetheories of tensor multiplication and decompositions are much developed, as well as eigenvaluesand singular values of tensors [22] [23] [19]. For applications, tensors appear in many fields,such as psychology [9], web searching [17] and so on. Although the notion of tensor transposeis often mentioned together with supersymmetric tensors [8], specific discussions concerningtensor transpose draw less attention.In this paper, tensors are viewed from a different prospective: tensor transpose. It is knownthat matrix transpose is a notion of a matrix and has several properties. Here, we focus ouremphases on extending the notion of transpose from 2-order tensors to high order tensors anddiscovering its properties. 1 a r X i v : . [ c s . NA ] N ov Ran Pan
Figure 1.1: 1-order, 2-order and 3-order tensorsIn Section 2, we will review knowledge of permutations and symmetry group [14], and usethem to define the tensor transpose. Then we discuss the classification and composition of tensortransposes. In Section 3, the relationship between transpose and tensor multiplication will bediscussed. It is known transpose of matrix multiplication satisfies that A T B T = ( BA ) T , where A and B are matrices. Proposition 3.1, as the most important result of this paper, will beintroduced. In section 4, we prove that tensor l p -eigenvalues are invariant under some certaintensor transpose. In Section 5, the discussion about the transpose and decomposition will becontinued. Two methods of decompositions CP decomposition and Tucker decomposition areconsidered separately in corresponding passages respectively. Before introducing the definition of tensor transpose, we review fundamental knowledge of per-mutations and symmetry group first.Assume set S = { , , , · · · , n } and σ is a permutation of S , usually, σ is denoted as follows,with σ ( i ) = k i σ = (cid:18) · · · nk k k · · · k n (cid:19) , where { k , k , k , · · · , k n } = { , , , · · · , n } = S .The same as functions, two permutations canbe composed. For example, σ and τ are two permutations and their composition τ σ ( i ) = τ ( σ ( i )).If there exist r different numbers i , i , · · · , i r subjected to σ ( i ) = i , σ ( i ) = i , · · · , σ ( i r − ) = i r , σ ( i r ) = i , the permutation σ is called a r-circle.All permutations are products of cycles. Specially, a 3-order permutation can be written inform of a cycle. For instance, σ = (cid:18) (cid:19) = (cid:0) (cid:1) . We denote σ − as the inverse of σ . Then we review the definition of a symmetry group. S isa finite set consisting of n elements. The set consisting of all permutations of any given set S , ensor Transpose and Its Properties S . The notation of symmetrygroup is S n . Apparently, there are n ! permutations for the set S .Now we can introduce the definition of tensor transpose. We know that matrix transpose isa permutation of the two indices. Considering the fact that a matrix is a 2-order tensor, in thispaper, we extend the definition of transpose to high order tensors. Definition 2.1
Let X be an n -order tensor. Y is called tensor transpose of X associated with σ , if entries Y ( i σ (1) , i σ (2) , . . . , i σ ( n ) ) = X ( i , i , . . . , i n ) , where σ is an element of S n but not anidentity permutation. Y is denoted by X T σ . This definition shows if X is of size I × I × · · · × I N , X T σ is of size I σ (1) × I σ (2) × · · · × I σ ( N ) .Elementwise, for example, assume X is a 3-order tensor and σ = (cid:0) (cid:1) ∈ S , i.e. σ (1) =2 , σ (2) = 3 , σ (3) = 1. We have X T σ ( i , i , i ) = X ( i , i , i ).Using Bader and Kolda’s MATLAB tensor toolbox [2], a tensor can be transposed in MAT-LAB. In tensor toolbox, the function PERMUTE is used to transpose a tensor.In order to facilitate the description and discussion, some special symbols are utilized fortranspose of 3-order tensors. positive transpose X T + = X T σ where σ = (cid:16) (cid:17) negative transpose X T − = X T σ where σ = (cid:16) (cid:17) first transpose X T = X T σ where σ = (cid:16) (cid:17) second transpose X T = X T σ where σ = (cid:16) (cid:17) third transpose X T = X T σ where σ = (cid:16) (cid:17) Considering a 3-order tensor is like a rectangular cuboid, transpose of a 3-order tensor has itsgeometric meaning. Transpose can be looked upon as a rotation of the rectangular cuboid.In terms of an n -order tensor, it is clear that a certain transpose is corresponding to a certainpermutation. Therefore, we’ve got the first property of tensor transpose in the paper. Proposition 2.2 An n -order tensor has n ! − different transposes. In addition, according to definition of transpose, we can define the supersymmetric tensor ina new way. Symmetry is an important notion for matrices, while it is called supersymmetry forhigh order tensors. In scientists’ previous work, a supersymmetric tensor is often described as atensor whose entries are invariant under any permutation of their indices. We know that matrix A is symmetric if A T = A . A similar definition of supersymmetric tensor is given as follows. Definition 2.3 X is called a supersymmetric tensor, if X = X T σ , for all σ ∈ S n , where n isthe order of X . From the viewpoint of group theory, tensor transpose can be treated as an action of a sym-metry group on a tensor and supersymmetric tensors can be treated as fixed elements of group S n .Besides, according to the definition of tensor transpose, tensor transposes can be classifiedinto two classes: total transpose and partial transpose. Ran Pan
Definition 2.4 X T σ is called a total transpose, if σ is a derangement. Otherwise, it’s a partialtranspose. A derangement [11] is a permutation such that none of the elements appear in theiroriginal position. It means σ is derangement if σ ( i ) (cid:54) = i , for all i ∈ { , , · · · , n } . Assume an n -order tensor has a n total transposes, apparently a n is also the number of derange-ments in symmetry group S n . We have a n = n ! n (cid:88) i =0 ( − i i ! , and it is called ”de Montmort number” [20]. For 3-order tensor X , a = 2, it has two totaltransposes X T + and X T − . And X T , X T and X T are partial transpose.It is known that two permutations can be composed, as well as two tensor transposes. Formatrix A , ( A T ) T = A . For high order tensors, we have following proposition. Proposition 2.5 ( X T σ ) T τ = X T ( τσ ) , where τ σ means the composition of the two permutations. From this proposition, it is obvious that the composition of transpose is equivalent to compo-sition of permutations. If τ = σ − , ( X T σ ) T σ − = X . Therefore, we have the property for 3-ordertensors. Corollary 2.6
Let X be a 3-order tensor. ( X T + ) T − = X , ( X T − ) T + = X , ( X T ) T = X , ( X T ) T = X , ( X T ) T = X . In this section, we will consider tensor transpose of tensor multiplication. High order tensormultiplication is much more complex than matrix multiplication, and high order tensors havemore transposes than matrices (i.e. 2-order tensors). A full treatment of tensor multiplicationcan be found in Bader and Kolda’s work [15] [1]. Here we only discuss some kinds multiplicationof them.
A familiar property of matrix transpose and matrix multiplication is that A T B T = ( BA ) T , where A and B are matrices. However, a high order tensor usually has more than three dimensions.Therefore, we must specify which dimension is multiplied by the matrix. In this passage, weadopt n -mode product [18].Let X be an I × I ×· · ·× I N tensor and U be a J n × I n matrix. Then the n -mode product of X and U is denoted by X × n U and its result is a tensor of size I × I ×· · ·× I n − × J n × I n +1 ×· · · × I N .The element of X × n U is defined as( X × n U )( i , . . ., i n − , j n , i n +1 , . . ., i N ) = I n (cid:88) i n =1 X ( i , i , . . ., i N ) U ( j n , i n ) . ensor Transpose and Its Properties Proposition 3.1
Let X be an N -order tensor, we have X T σ × σ − ( n ) U = ( X × n U ) T σ . Proof:Let Y = X T σ , Z = Y × σ − ( n ) U , P = X × ( n ) U , Q = P T σ .According to the definition of tensor transpose, we have Y ( i , i , . . . , i N ) = X ( i σ − (1) , i σ − (2) , . . . , i σ − ( N ) ) , and Z ( i , . . ., i n − , j n , i n +1 , . . ., i N ) = I σ − n ) (cid:88) i σ − n ) =1 Y ( i , i , . . ., i N ) U ( j σ − ( n ) , i σ − ( n ) ) , that is Z ( i , . . ., i n − , j n , i n +1 , . . ., i N ) = I σ − n ) (cid:88) i σ − n ) =1 X ( i σ − (1) , i σ − (2) , . . . , i σ − ( N ) ) U ( j σ − ( n ) , i σ − ( n ) ) . For the right side of the equation, P ( i , . . ., i n − , j n , i n +1 , . . ., i N ) = I n (cid:88) i n =1 X ( i , i , . . ., i N ) U ( j n , i n ) , since Q = P T σ , Q ( i , i , . . . , i N ) = P ( i σ − (1) , i σ − (2) , . . . , i σ − ( N ) ) , Q ( i , . . ., i n − , j n , i n +1 , . . ., i N ) = I σ − n ) (cid:88) i σ − n ) =1 X ( i σ − (1) , i σ − (2) , . . . , i σ − ( N ) ) U ( j σ − ( n ) , i σ − ( n ) ) . Therefore, Z ( i , . . ., i n − , j n , i n +1 , . . ., i N ) = Q ( i , . . ., i n − , j n , i n +1 , . . ., i N ). (cid:4) In fact, Proposition 3.1 is an extension of 2-order situation: A T B T = ( BA ) T . For matrices, A × B = BA , A × B = AB T . According to proposition 3.1, ( BA ) T = ( A × B ) T = A T × B = A T B T . We consider three general scenarios for tensor-tensor multiplication: outer product, contractedproduct, and inner product [1].For outer product and contracted product, tensor transposes of them do not have significantcharacteristics.For the inner product of two tensors, it requires that these two tensors are of the same size.Assume A , B are two tensors of size I × I × · · · × I N , the inner product of A , B is given by (cid:104) A , B (cid:105) = I (cid:88) i =1 I (cid:88) i =1 · · · I N (cid:88) i N =1 A ( i , i , . . . , i N ) B ( i , i , . . . , i N ) Ran Pan
Proposition 3.2 A and B are two tensors of the same size, we have (cid:104) A T σ , B T σ (cid:105) = (cid:104) A , B (cid:105) . This property follows the definition of the inner product directly.Using the inner product, the Frobenius norm of a tensor is given by (cid:107) A (cid:107) F = (cid:112) (cid:104) A , A (cid:105) . Thenwe have the following property. Corollary 3.3 (cid:107) A T σ (cid:107) F = (cid:107) A (cid:107) F It is known that eigenvalues keep invariant after the matrix is transposed. There are similarsituations for eigenvalues of transposed tensors. In this section, we adopt Lim’s definition of l p -eigenvalues of nonsymmetric tensors [19]. A ∈ R n × n ×···× n is a K -order tensor. The homogeneous polynomial associated with tensor A can be conveniently expressed as A ( x , · · · , x ) := A × x · · · × K x . Since A has K sides, the tensor has K different forms of eigenpairs as follows A ( I n , x , · · · , x ) = λ ϕ p − ( x ) A ( x , I n , · · · , x ) = λ ϕ p − ( x ) · · · A ( x K , x K , · · · , I n ) = λ K ϕ p − ( x K )where I n is an n -by- n identity matrix, ϕ p − ( x ) := [ sgn ( x ) | x | p , · · · , sgn ( x n ) | x n | p ], and sgn ( x )is the sign function. The unit vector x i is called mode- i eigenvector of A corresponding to themode- i eigenvalue λ i , i = 1 , , · · · , K . Proposition 4.1
The mode-i eigenpairs are invariant under tensor transopose associated with σ , if σ ( i ) = i . Proof:Here, we take mode-1 as an example. A ∈ R n × n ×···× n is a K -order tensor. Let B = A T σ , where σ (1) = 1.We have B ( I n , x , · · · , x ) = B × x × x · · · × K x = A T σ × x × x · · · × K x According to Proposition 3.1 X T σ × n U = ( X × σ ( n ) U ) T σ and σ (1) = 1, A T σ × x × x · · · × K x = ( A × σ (2) x ) T σ × x · · · × K x = ( A × σ (2) x × σ (3) x ) T σ · · · × K x · · · = ( A × σ (2) x × σ (3) x · · · × σ ( K ) x ) T σ = ( A × x × x · · · × K x ) T σ = A × x × x · · · × K x ensor Transpose and Its Properties B ( I n , x , · · · , x ) = A ( I n , x , · · · , x ). When σ (1) = 1, A T σ has the same eigenpairswith A . (cid:4) In [6], Zhen Chen , Lin-zhang Lu and Zhi-bing Liu proposed and proved a similar property asProposition 4.1 by a different method.
There are a number of tensor decompositions among which CANDECOMP/PARAFAC decom-position and Tucker decomposition are most popular [16] [7]. They are used in psychometrics,applied statistics, weblink analysis and many other fields. In this section, we will focus on therelationship between transpose and the two major decompositions.
CP decomposition is short for CANDECOMP/PARAFAC decomposition, which are introducedby Hitchcock [13] [12], Cattell [4] [5], Carroll and Chang [3], and Harshman [10]. The CPdecomposition is strongly linked with rank-one tensors. Usually, an n -order rank-one tensor canbe written in the outer product of n vectors. For example, X ∈ R I × I × I , X = u ◦ u ◦ u , then X is a rank-one tensor, where u , u , u are vectors and ◦ is outer product operator. Proposition 5.1
If rank-one tensor X = u ◦ u ◦ · · · ◦ u N , X T σ = u σ (1) ◦ u σ (2) ◦ · · · ◦ u σ ( N ) . Proof:Assume X ∈ R I × I ×···× I N ,then X = u ◦ u ◦ · · · ◦ u N that is X ( i , i , . . . , i N ) = u ( i ) u ( i ) · · · u N ( i N ) , and X T σ ( i , i , . . . , i N ) = X ( i σ − (1) , i σ − (2) , . . . , i σ − ( N ) ) = u ( i σ − (1) ) u ( i σ − (2) ) · · · u N ( i σ − ( N ) )because for all k ∈ { , , · · · , N } , there exists j , s.t. σ ( j ) = k, σ − ( k ) = j so u k ( i σ − ( k ) ) = u σ ( k ) ( i j )therefore u ( i σ − (1) ) u ( i σ − (2) ) · · · u N ( i σ − ( N ) ) = u σ (1) ( i ) u σ (2) ( i ) · · · u σ ( N ) ( i N )that is X T σ = u σ (1) ◦ u σ (2) ◦ · · · ◦ u σ ( N ) . (cid:4) Ran Pan
The CP decomposition factorizes a tensor into a sum of rank-one tensors. Take 3-ordersituation as an example. Let X be a 3-order tensor, and X i be rank-one 3-order tensors, X i = a i ◦ b i ◦ c i . Then the CP decomposition of X can be written as X = R (cid:88) i =1 X i = R (cid:88) i =1 a i ◦ b i ◦ c i We denote matrix A , A , A , as the combination of vectors a i , b i , c i , i.e., A = (cid:0) a a · · · a R (cid:1) .Then CP decomposition can be expressed by X = R (cid:88) i =1 a i ◦ b i ◦ c i := [[ A , A , A ]]According to Proposition 5.1, we have a property as follows, Proposition 5.2 If X = [[ A , A , A ]] , then X T σ = [[ A σ (1) , A σ (2) , A σ (3) ]] . The rank of a tensor is defined as the smallest number of rank-one tensors that exactly sumup to that tensor. From previous research [16], rank decompositions are often unique. Thenwe have the following property.
Proposition 5.3
The rank of a certain tensor is invariant under any transpose.
The Tucker decomposition was first introduced by Tucker [24] [25] and it decomposes a tensorinto a core tensor multiplied by a matrix along each mode. Here, we consider the 3-ordersituation. Let X be a 3-order tensor of size I × I × I , we have X = G × A × A × A = J (cid:88) j J (cid:88) j J (cid:88) j g j j j a j ◦ b j ◦ c j := [[ G ; A , A , A ]] , where A i ∈ R I i × J i are the factor matrices. The tensor G of size J × J × J is called the core tensorof X . Proposition 5.4 If X = [[ G ; A , A , A ]] , X T σ = [[ G T σ ; A σ (1) , A σ (2) , A σ (3) ]] . Proof:According to Proposition 3.1, we have X T σ = ( G × A × A × A ) T σ = ( G × A × A ) T σ × σ − (3) A = G T σ × σ − (1) A × σ − (2) A × σ − (3) A so X T σ = G T σ × σ − (1) A × σ − (2) A × σ − (3) A ensor Transpose and Its Properties G × A × A = G × σ (1) A σ (1) × σ (2) A σ (2) , seen in [1] , we have G T σ × σ − (1) A × σ − (2) A × σ − (3) A = G T σ × A σ (1) × A σ (2) × A σ (3) , therefore X T σ = [[ G T σ ; A σ (1) , A σ (2) , A σ (3) ]] . (cid:4) From this property, we can see that G T σ is the core tensor of X T σ , if G is the core tensor of X .Actually, Proposition 5.2, 5.3, 5.4 also apply to 4 or higher order tensors. The notion of tensor transpose is often mentioned together with supersymmetric tensors butspecific discussions concerning tensor transpose draw less attention. In this paper, we proposethe definition of tensor transpose and proved some basic properties. According to Proposition3.1, properties regarding inner product, tensor eigenvalues, tensor decompositions and rankare derivated in following sections. In future work, the introduction of tensor transpose maybe useful for tensor theory research, computation or algorithms improvement. We will keepworking on it.
References [1] B. W. Bader and T. G. Kolda,
Algorithm 862: MATLAB tensor classes for fast algorithmprototyping , ACM Trans. Math. Software, 32 (2006), pp. 635–653.[2] B. W. Bader and T. G. Kolda,
MATLAB Tensor Toolbox, Version 2.2. , Available athttp://csmr.ca.sandia.gov/ ∼ tgkolda/TensorToolbox/, (2007).[3] J. D. Carroll and J. J. Chang, Analysis of individual differences in multidimensional scalingvia an N-way generalization of Eckart-Young decomposition,
Psychometrika, 35 (1970), pp.283–319.[4] R. B. Cattell,
Parallel proportional profiles and other principles for determining the choiceof factors by rotation , Psychometrika, 9 (1944), pp. 267–283.[5] R. B. Cattell,
The three basic factor-analytic research designstheir interrelations and deriva-tives , Psych. Bull., 49 (1952), pp. 452–499.[6] Z. Chen, L. Lu, Z. Liu
The eigenvalue problems for tensor and tensor transposition (Chi-nese),
Journal of Xiamen University (Natural Science), Vol. 51, No. 3, (2012)0
Ran Pan [7] P. Comon,
Tensor decompositions: State of the art and applications , Mathematics in SignalProcessing V, J. G. McWhirter and I. K. Proudler, eds., Oxford University Press, (2001),pp. 1–24.[8] P. Comon, G. Golub, L.H. Lim and B. Mourrain
Symmetric tensors and symmetric tensorrank , SIAM J. Matrix Anal. Appl., 30, (2008), pp. 1254–1279.[9] S. C. Deerwester, S. T. Dumais, T. K. Landauer, G. W. Furnas, and R. A. Harshman,
Indexing by latent semantic analysis,
J. Amer. Soc. Inform. Sci., 41 (1990), pp. 391–407.[10] R. A. Harshman,
Foundations of the PARAFAC procedure: Models and conditions for an”explanatory” multi-modal factor analysis , UCLA Working Papers in Phonetics, 16 (1970),pp. 1–84.[11] M. Hassani,
Derangements and Applications , J. Integer Seq. 6, No. 03.1.2, (2003), pp. 1–8.[12] F. L. Hitchcock,
Multilple invariants and generalized rank of a p-way matrix or tensor , J.Math. Phys., 7 (1927), pp. 39–79.[13] F. L. Hitchcock,
The expression of a tensor or a polyadic as a sum of products , J.Math.Phys., 6 (1927), pp. 164–189.[14] N. Jacobson,
Basic Algebra(I) (2nd Edition) , New York: W.H. Freeman and Company,(1985).[15] T. G. Kolda,
Multilinear Operators for Higher-Order Decompositions , Tech. ReportSAND2006–2081, Sandia National Laboratories, Albuquerque, NM, Livermore, CA, (2006).[16] T. G. Kolda and B. W. Bader,
Tensor Decompositions and Applications , SIAM 2009: Vol.51, No. 3, pp. 455–500.[17] T. G. Kolda and B. W. Bader,
The TOPHITS model for higher-order web link analysis, inWorkshop on Link Analysis, Counterterrorism and Security, (2006).[18] L. de Lathauwer, B. de Moor, and J. Vandewalle,
A multilinear singular value decomposi-tion , SIAM J. Matrix Anal. Appl., 21, (2000), pp. 1253–1278.[19] L.H. Lim,
Singular values and eigenvalues of tensors: A variational approach,
Proceed-ings of the 1st IEEE International Workshop on Computational Advances in Multi-SensorAdaptive Processing (CAMSAP), December 13–15, (2005), pp. 129–132.[20] P. R. de Montmort,
Essay d’analyse sur les jeux de hazard. Paris: Jacque Quillau. SecondeEdition , Revue augmente de plusieurs Lettres. Paris: Jacque Quillau. (1713).[21] L. Qi,
Eigenvalues and invariants of tensors , J. Math. Anal. Appl. 325 (2007), pp. 1363–1377[22] L. Qi,
Eigenvalues of a real supersymmetric tensor,
J. Symbolic Comput., 40 (2005), pp.1302–1324.[23] L. Qi,
Rank and eigenvalues of a supersymmetric tensor, the multivariate homogeneouspolynomial and the algebraic hypersurface it defines,
J. Symbolic Comput., 41 (2006), pp.1309–1327. ensor Transpose and Its Properties
Implications of factor analysis of three-way matrices for measurement ofchange, in Problems in Measuring Change, C. W. Harris, ed., University of WisconsinPress, (1963), pp. 122–137.[25] L. R. Tucker,