Real double flag varieties for the symplectic group
aa r X i v : . [ m a t h . R T ] M a r REAL DOUBLE FLAG VARIETIES FOR THE SYMPLECTIC GROUP
KYO NISHIYAMA AND BENT ØRSTED
Abstract.
In this paper we study a key example of a Hermitian symmetric space anda natural associated double flag variety, namely for the real symplectic group G and thesymmetric subgroup L , the Levi part of the Siegel parabolic P S . We give a detailedtreatment of the case of the maximal parabolic subgroups Q of L corresponding toGrassmannians and the product variety of G/P S and L/Q ; in particular we classify the L -orbits here, and find natural explicit integral transforms between degenerate principalseries of L and G . Introduction
The geometry of flag varieties over the complex numbers, and in particular doubleflag varieties, have been much studied in recent years (see, e.g., [FN16], [HT12], [Tra09],[FGT09] etc.). In this paper we focus on a particular case of a real double flag varietywith the purpose of understanding in detail (1) the orbit structure under the naturalaction of the smaller reductive group (2) the construction of natural integral transformsbetween degenerate principal series representations, equivariant for the same group. Eventhough aspects of (1) are known from general theory (e.g., [KM14], [KO13] and referencestherein), the cases we treat here provide new and explicit information; and for (2) wealso find new phenomena, using the theory of prehomogeneous vector spaces and relativeinvariants. In particular the Hermitian case we study has properties complementary toother well-known cases of (2). For this, we refer the readers to [KS15], [MØO16], [KØP11],[CKØP11], [Zha09], [BSKZ14] among others.Thus in this paper we study a key example of a Hermitian symmetric space and anatural associated double flag variety, namely for the real symplectic group G and thesymmetric subgroup L , the Levi part of the Siegel parabolic P S . We give a detailedtreatment of the case of the maximal parabolic subgroup Q of L corresponding to Grass-mannians and the product variety of G/P S and L/Q ; in particular we classify the open L -orbits here, and find natural explicit integral transforms between degenerate principalseries of L and G . We realize these representations in their natural Hilbert spaces anddetermine when the integral transforms are bounded operators. As an application wealso obtain information about the occurrence of finite-dimensional representations of L Date : Ver. 0.95 [2017/03/12 10:21:51] (compiled on March 21, 2017).2010
Mathematics Subject Classification.
Primary 22E46; Secondary 14M15, 11S90, 22E45, 47G10.
Key words and phrases. double flag variety, Hermitian symmetric space, prehomogeneous vector space,degenerate principal series representation, integral kernel operator.K. N. is supported by JSPS KAKENHI Grant Numbers in both of these generalized principal series representations of G resp. L . It follows fromgeneral principles, that our integral transforms, depending on two complex parametersin certain half-spaces, may be meromorphically continued to the whole parameter space;and that the residues will provide kernel operators (of Schwartz kernel type, possibly evendifferential operators), also intertwining (i.e., L -equivariant). For general background onintegral operators depending meromorphically on parameters, and for equivariant integraloperators – introduced by T. Kobayashi as symmetry-breaking operators – as we studyhere, see [KS15], [MØO16] and [KK79]. However, we shall not pursue this aspect here,and it is our future subject.It will be clear, that the structure of our example is such that other Hermitian groups, inparticular of tube type, will be amenable to a similar analysis; thus we contend ourselveshere to give all details for the symplectic group only.Let us fix notations and explain the content of this paper more explicitly. So let G =Sp n ( R ) be a real symplectic group. We denote a symplectic vector space of dimension2 n by V = R n with a natural symplectic form defined by h u, v i = t u J n v , where J n = (cid:18) − n n (cid:19) . Thus, our G is identified with Sp( V ). Let V + = span R { e , e , . . . , e n } spanned by the first n fundamental basis vectors, which is a Lagrangian subspace of V .Similarly, we put V − = span R { e n , e n +1 , . . . , e n } , a complementary Lagrangian subspaceto V + , and we have a complete polarization V = V + ⊕ V − . The Lagrangians V + and V − are dual to each other by the symplectic form, so that we can and often do identify V − = ( V + ) ∗ .Let P S = Stab G ( V + ) = { g ∈ G | gV + = V + } be the stabilizer of the Lagrangiansubspace V + . Then P S is a maximal parabolic subgroup of G with Levi decomposition P S = L ⋉ N , where L = Stab G ( V + ) ∩ Stab G ( V − ), the stabilizer of the polarization,and N is the unipotent radical of P S . We call P S a Siegel parabolic subgroup. Since G = Sp( V ) acts on Lagrangian subspaces transitively, Λ := G/P S is the collection of allLagrangian subspaces in V . We call this space a Lagrangian flag variety and also denoteit by LGr( R n ).The Levi subgroup L of P S is explicitly given by L = n(cid:18) a t a − (cid:19) (cid:12)(cid:12)(cid:12) a ∈ GL n ( R ) o ≃ GL n ( R ) , and we consider it to be GL( V + ) which acts on V − = ( V + ) ∗ in the contragredient manner.The unipotent radical N of P S is realized in the matrix form as N = n(cid:18) z (cid:19) (cid:12)(cid:12)(cid:12) z ∈ Sym n ( R ) o ≃ Sym n ( R )via the exponential map. Note that (cid:18) a b t a − (cid:19) ∈ P S if and only if a t b ∈ Sym n ( R ),which in turn equivalent to a − b ∈ Sym n ( R ). EAL DOUBLE FLAG VARIETIES 3
Take a maximal parabolic subgroup Q in L = GL( V + ) which stabilizes d -dimensionalisotropic space U ⊂ V + . Then Ξ d := L/Q = Gr d ( V + ) = Gr d ( R n ) is the Grassmannian of d -dimensional spaces. Note that, in the standard realization, Q = P GL( d,n − d ) = n(cid:18) α ξ β (cid:19) (cid:12)(cid:12)(cid:12) α ∈ GL d ( R ) , β ∈ GL n − d ( R ) , ξ ∈ M d,n − d ( R ) o . Now, our main concern is a double flag variety X = Λ × Ξ d = G/P S × L/Q on which L = GL n ( R ) acts diagonally. We are strongly interested in the orbit structure of X underthe action of L and its applications to representation theory. Goal and Main Results 0.1.
We will consider the following problems.(1) To prove there are finitely many L -orbits on the double flag variety X = Λ × Ξ d .We will give a complete classification of open orbits, and recursive strategy to determinethe whole structure of L -orbits on X . See Theorems 2.7 and 4.3.(2) To construct relative invariants on each open orbits. We will use them to defineintegral transforms between degenerate principal series representations of L and that of G . For this, see §
7, especially Theorems 7.1 and 7.2.Here we will make a short remark on the double flag varieties over the complex numberfield (or, more correctly, over an algebraically closed fields of characteristic zero).Let us complexify everything which appears in the setting above, so that G C = Sp n ( C )and L C ≃ GL n ( C ). The complexifications of the parabolics are P S, C , the stabilizer of aLagrangian subspace in the symplectic vector space C n , and Q C , the stabilizer of a d -dimensional vector space in C n . Then it is known that the double flag variety X C = G C /P S, C × L C /Q C has finitely many L C -orbits or Q C \ G C /P S, C < ∞ . In this case, onecan replace the maximal parabolic Q C by a Borel subgroup B L, C of L C , and still there arefinitely many L C orbits in G C /P S, C × L C /B L, C (see [NO11] and [HNOO13, Table 2]).Even if there are only finitely many orbits of a complex algebraic group, say L C , actingon a smooth algebraic variety, there is no guarantee for finiteness of orbits of real formsin general . So our problem over reals seems impossible to be deduced from the resultsover C .On the other hand, in the case of the complex full flag varieties, there exists a famousbijection between K C orbits and G R orbits called Matsuki correspondence [Mat88]. Bothorbits are finite in number. In the case of double flag varieties, there is no such knowncorrespondences. It might be interesting to pursue such correspondences.Toshiyuki Kobayashi informed us that the finiteness of orbits X/L < ∞ also followsfrom general results on visible actions [Kob05]. We thank him for his kind notice. Acknowledgement.
K. N. thanks Arhus University for its warm hospitality during thevisits in August 2015 and 2016. Most of this work has been done in those periods. It is known that there is a canonical bijection L ( R ) \ ( L/H )( R ) = ker( H ( C ; H ) → H ( C ; L )), where C = Gal( C / R ) and H ( C ; H ) denotes the first Galois cohomology group. See [BJ06, Eq. (II.5.6)]. KYO NISHIYAMA AND BENT ØRSTED Elementary properties of G = Sp n ( R )In this section, we will give very well known basic facts on the symplectic group for thesake of fixing notations. We define G = Sp n ( R ) = { g ∈ GL n ( R ) | t g J n g = J n } where J n = (cid:16) − n n (cid:17) . The following lemmas are quite elementary and well known. We just present them becauseof fixing notations.
Lemma 1.1.
If we write g = (cid:16) a bc d (cid:17) ∈ GL n ( R ) , then g belongs to G if and only if t a c, t b d ∈ Sym n ( R ) and t a d − t c b = 1 .Proof. We rewrite t g J g = J by coordinates, and get t c a − t a c = 0 t c b − t a d = − t d a − t b c = 1 t d b − t b d = 0which shows the lemma. (cid:3) Lemma 1.2.
If we write g = (cid:16) a bc d (cid:17) ∈ G , then g − = (cid:16) t d − t b − t c t a (cid:17) .Proof. Since t g J g = J , we get g − = J − t g J = − J t g J = (cid:16) t d − t b − t c t a (cid:17) . (cid:3) Lemma 1.3.
If we write g = (cid:16) a bc d (cid:17) ∈ G and p = (cid:16) x z y (cid:17) ∈ P S , then g − pg = (cid:18) t d xa + t d zc − t b yc t d xb + t d zd − t b yd − t c xa − t c zc + t a yc − t c xb − t c zd + t a yd (cid:19) . (1.1) Note that, in fact, y = t x − .Proof. Just a calculation, using Lemma 1.2. (cid:3)
A maximal compact subgroup K of G is given by K = Sp n ( R ) ∩ O(2 n ). Lemma 1.4.
An element g = (cid:16) a bc d (cid:17) ∈ G belongs to K if and only if b = − c, d = a, t a b ∈ Sym n ( R ) , and t a a + t b b = 1 n hold. Consequently, K = (cid:26)(cid:18) a b − b a (cid:19) (cid:12)(cid:12)(cid:12) a + ib ∈ U( n ) (cid:27) . (1.2) EAL DOUBLE FLAG VARIETIES 5
Proof. g ∈ G belongs to O(2 n ) if and only if t g = g − . From Lemma 1.2, we get a = d and c = − b . From 1.1, we get the rest two equalities.Note that if a + ib ∈ U( n )( a + ib ) ∗ ( a + ib ) = ( t a − i t b ) ( a + ib )= ( t a a + t b b ) + i ( t a b − t b a ) = 1 n . This last formula is equivalent to the above two equalities. (cid:3) L -orbits on the Lagrangian flag variety ΛNow, let us begin with the investigation of L orbits on Λ = G/P S , which should bewell-known.Let us denote the Weyl group of P S by W P S , which is isomorphic to S n , the symmetricgroup of n -th order. In fact, it coincides with the Weyl group of L .By Bruhat decomposition, we have G/P S = [ w ∈ W G P S wP S /P S = G w ∈ W PS \ W G /W PS P S wP S /P S , (2.1)where in the second sum w moves over the representatives of the double cosets. Thedouble coset space W P S \ W G /W P S ≃ S n \ ( S n ⋉ ( Z / Z ) n ) /S n has a complete system ofrepresentatives of the form { w k = (1 , . . . , , − , . . . , −
1) = (1 k , ( − n − k ) | ≤ k ≤ n } ⊂ ( Z / Z ) n . We realize w k in G as w k = − n − k k n − k k . (2.2) Lemma 2.1.
For ≤ k ≤ n , we temporarily write w = w k . Then P S wP S /P S = w ( w − P S w ) P S /P S ≃ w − P S w/ ( w − P S w ∩ P S ) contains N w given below as an open densesubset. N w = (cid:26)(cid:18) n η n (cid:19) (cid:12)(cid:12)(cid:12) η = (cid:18) ζ ξ t ξ k (cid:19) , ζ ∈ Sym n − k ( R ) , ξ ∈ M n − k,k ( R ) (cid:27) (2.3) KYO NISHIYAMA AND BENT ØRSTED
Proof.
Take (cid:16) x z y (cid:17) ∈ P S and write w = (cid:16) a bc d (cid:17) , where a = d = (cid:16) k (cid:17) , c = − b = (cid:16) n − k
00 0 (cid:17) . Then, using the formula in Lemma 1.3, we can calculate as w − (cid:16) x z y (cid:17) w = (cid:18) t d xa + t d zc − t b yc t d xb + t d zd − t b yd − t c xa − t c zc + t a yc − t c xb − t c zd + t a yd (cid:19) (2.4)= (cid:18) x + z + y − x + z + y − x − z + y x − z + y (cid:19) (2.5)= y y z x − x z − z − x x − z y y (2.6)Let us rewrite the last formula in the form (cid:16) η (cid:17)(cid:16) α β δ (cid:17) = (cid:16) α βηα ηβ + δ (cid:17) , so that we get η = (cid:18) − z − x y (cid:19) (cid:18) y z x (cid:19) − = (cid:18) − z − x y (cid:19) (cid:18) y − − x − z y − x − (cid:19) = (cid:18) − z y − + x x − z y − − x x − y y − (cid:19) , (2.7)provided that y − and x − exist (an open condition). Note that we can take z and z arbitrary, and also that, if we put x = 0 and y = 0, we can take x (which determines y ) arbitrary. This shows the last formula (2.7) above exhausts η of the form in (2.3). (cid:3) Remark 2.2. The formula (2.7) actually gives a symmetric matrix. One can check thisdirectly, using y = t x − . See also Lemma 2.4 below.Let us consider L = GL n ( R ) action on the k -th Bruhat cell P S w k P S /P S . It is justthe left multiplication. However, if we identify it with w − P S w/ ( w − P S w ) ∩ P S as inLemma 2.1, the action of a ∈ L is given by the left multiplication of w − aw . Thisconjugation is explicitly given as w − aw = h ′ h ′ h − h − h h h ′ h ′ where a = (cid:16) h t h − (cid:17) , h = (cid:16) h h h h (cid:17) , t h − = h ′ = (cid:16) h ′ h ′ h ′ h ′ (cid:17) , (2.8) EAL DOUBLE FLAG VARIETIES 7 which can be read off from Equation (2.6).
Lemma 2.3.
There are exactly (cid:0) n − k +22 (cid:1) of L -orbits on the Bruhat cell P S w k P S /P S (0 ≤ k ≤ n ) . A complete representatives of L -orbits is given as n(cid:16) z (cid:17) w k P S /P S (cid:12)(cid:12)(cid:12) z = (cid:16) I r,s
00 0 (cid:17) ∈ Sym n ( R ) , ≤ r + s ≤ n − k o , where I r,s = diag(1 r , − s ) .Proof. For the brevity, we will write w = w k . Firstly, we observe that by the left multi-plication of L clearly we can choose orbit representatives from the set { (cid:16) z (cid:17) wP S /P S | z ∈ Sym n ( R ) } . Then, by the calculations in the proof of Lemma 2.1 and Equation (2.7), it reduces to thesubset (cid:26)(cid:16) n η n (cid:17) (cid:12)(cid:12)(cid:12)(cid:12) η = (cid:16) ζ
00 0 (cid:17) , ζ ∈ Sym n − k ( R ) (cid:27) ⊂ w − P S w/ ( w − P S w ) ∩ P S . (2.9)Now let us consider the action of L on this set. Take a = (cid:16) h t h − (cid:17) ∈ L , where h = diag( h , k ) ( h ∈ GL n − k ( R )). Then, the action of a is the left multiplication of w − aw as explained above (see Equation (2.8)). As a consequence, it brings to( w − aw ) (cid:16) η (cid:17) P S /P S = h ′
00 1 h ζ h
00 0 0 1 P S /P S = h ζ t h P S /P S . Now it is well known that for a suitable choice of h ∈ GL n − k ( R ), we get h ζ t h = (cid:16) I r,s
00 0 (cid:17) for a certain signature ( r, s ) with r + s ≤ n − k . (cid:3) Let us explicitly describe the L -action on N w ⊂ ( w − P S w ) / ( w − P S w ) ∩ P S . Lemma 2.4.
The action of w − aw in Equation (2.8) on (cid:16) n n η n (cid:17) ∈ N w , η = (cid:16) ζ ξ t ξ k (cid:17) ( ζ ∈ Sym n − k ( R ) , ξ ∈ M n − k,k ( R )) , is given by η = (cid:16) ζ ξ t ξ (cid:17) a w · η = (cid:16) A B t B (cid:17) where ( A = ( h + Bh ) ζ t ( h + Bh ) ,B = ( − h + h ξ )( h − h ξ ) − . So the action on ξ -part is linear fractional, while action on ζ -part is a mixture of unimod-ular and linear fractional action. KYO NISHIYAMA AND BENT ØRSTED
Proof.
Take a ∈ L as in Equation (2.8) and we use the formula of w − aw there. w − aw n − k
00 1 k ζ ξ n − k t ξ k = h ′ h ′ h − h − h h h ′ h ′ n − k
00 1 k ζ ξ n − k t ξ k = h ′ + h ′ t ξ h ′ − h ζ h − h ξ − h h ζ − h + h ξ h h ′ + h ′ t ξ h ′ =: (cid:16) η (cid:17)(cid:16) α β δ (cid:17) = (cid:16) α βηα ηβ + δ (cid:17) . From this, we calculate η = (cid:18) h ζ − h + h ξh ′ + h ′ t ξ (cid:19) (cid:18) h ′ + h ′ t ξ − h ζ h − h ξ (cid:19) − = (cid:18) h ζ − h + h ξh ′ + h ′ t ξ (cid:19) (cid:18) ( h ′ + h ′ t ξ ) − h − h ξ ) − h ζ ( h ′ + h ′ t ξ ) − ( h − h ξ ) − (cid:19) =: (cid:16) A BC (cid:17) , where A = h ζ ( h ′ + h ′ t ξ ) − + ( − h + h ξ )( h − h ξ ) − h ζ ( h ′ + h ′ t ξ ) − ,B = ( − h + h ξ )( h − h ξ ) − ,C = ( h ′ + h ′ t ξ )( h ′ + h ′ t ξ ) − . We will rewrite these formulas neatly.Firstly, we notice it should hold B = t C . Let us check it. For this, we compare h (cid:16) − ξ (cid:17) and t h − (cid:16) t ξ (cid:17) . Using notation h ′ = t h − , we calculate both as h (cid:16) − ξ (cid:17) = (cid:16) h h h h (cid:17)(cid:16) − ξ (cid:17) = (cid:18) h − h ξ + h h − h ξ + h (cid:19) (2.10) t h − (cid:16) t ξ (cid:17) = h ′ (cid:16) t ξ (cid:17) = (cid:16) h ′ h ′ h ′ h ′ (cid:17)(cid:16) t ξ (cid:17) = (cid:18) h ′ + h ′ t ξ h ′ h ′ + h ′ t ξ h ′ (cid:19) ∴ taking transpose, (cid:16) ξ (cid:17) h − = (cid:18) t h ′ + ξ t h ′ t h ′ + ξ t h ′ t ξ t h ′ t h ′ (cid:19) (2.11) EAL DOUBLE FLAG VARIETIES 9
Since (2.10) and (2.11) are mutually inverse, we get( t h ′ + ξ t h ′ ) h + ( t h ′ + ξ t h ′ t ξ ) h = 1 n − k , (2.12)( t h ′ + ξ t h ′ )( − h ξ + h ) + ( t h ′ + ξ t h ′ t ξ )( − h ξ + h ) = 0 , (2.13) t h ′ h + t h ′ h = 0 , (2.14) t h ′ ( − h ξ + h ) + t h ′ ( − h ξ + h ) = 1 k , (2.15)and taking transpose of Equation (2.12), t h ( h ′ + h ′ t ξ ) + t h ( h ′ + h ′ t ξ ) = 1 n − k . (2.16)Now, we calculate t C = t h ( h ′ + h ′ t ξ )( h ′ + h ′ t ξ ) − i = ( t h ′ + ξ t h ′ ) − ( t h ′ + ξ t h ′ t ξ ) = − ( − h ξ + h )( − h ξ + h ) − = B, where in the last equality we use Equation (2.13). This also proves the formula for linearfractional action on ξ .Secondly, we check that A is symmetric. A = h ζ ( h ′ + h ′ t ξ ) − + Bh ζ ( h ′ + h ′ t ξ ) − = ( h + Bh ) ζ ( h ′ + h ′ t ξ ) − = ( h + Bh ) ζ h t h + t h ( h ′ + h ′ t ξ )( h ′ + h ′ t ξ ) − i (by Eq. (2.16))= ( h + Bh ) ζ ( t h + t h C )= ( h + Bh ) ζ t ( h + Bh ) . ( ∵ C = t B )This proves that A is symmetric and at the same time the formula of the action on ζ inthe lemma. (cid:3) Lemma 2.5.
For a representative p ( k ; r,s ) := n − k
00 1 k ζ n − k
00 0 0 1 k , ζ = (cid:16) I r,s
00 0 (cid:17) ∈ Sym n − k ( R ) of L -orbits in the k -th Bruhat cell w − P S w/ ( w − P S w ) ∩ P S (see Lemma 2.3), the stabilizeris given by Stab L ( p ( k ; r,s ) ) = n a = (cid:16) h t h − (cid:17) (cid:12)(cid:12)(cid:12) h = (cid:16) h h h (cid:17) ∈ GL n ( R ) , h ζ t h = ζ o . (2.17) Thus an orbit O ( k ; r,s ) through p ( k ; r,s ) is isomorphic to GL n ( R ) /H ( k ; r,s ) , where H ( k ; r,s ) is thecollection of h given in Equation (2.17) .Proof. We put ξ = 0 in Lemma 2.4, and assume that a w · η = η . It gives B = − h h − = 0and A = h ζ t h = ζ . Here, we assume h is regular. So, under this hypothesis, we get h = 0. Since the stabilizer is a closed subgroup, we must have h = 0 in any case (as amatter of fact, actually h must be regular). (cid:3) For the later reference, we reinterpret the above lemma by Lagrangian realization.Recall that
G/P S is isomorphic to the set of Lagrangian subspaces in V denoted as Λ.The isomorphism is explicitly given by G/P S ∋ gP S g · V + ∈ Λ, here we identify V + with the space span R { e , . . . , e n } spanned by the first n fundamental vectors in V = R n .For v = P ni =1 c i e i ∈ V + , we denote v ( n − k ) = P n − ki =1 c i e i and v ( k ) = P kj =1 c n − k + j e n − k + j sothat v = v ( n − k ) + v ( k ) . Lemma 2.6.
With the notation introduced above, L -orbits on the Lagrangian Grassman-nian Λ ≃ G/P S has a representatives of the following form. V ( k ; r,s ) = ( u = − ζ v ( n − k ) v ( k ) v ( n − k ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) v ∈ V + ) , where ζ = (cid:16) I r,s
00 0 (cid:17) .Here ≤ k ≤ n and r, s ≥ denote the signature which satisfy ≤ r + s ≤ k .Proof. By Lemma 2.5, we know the representatives of L -orbits in the k -th Bruhat cell w − P S w/ ( w − P S w ) ∩ P S ( w = w k ). They are denoted as p ( k ; r,s ) . The correspondingLagrangian subspace is obtained by w k p ( k ; r,s ) · V + . If we take v ∈ V + and write it as v = v ( n − k ) + v ( k ) as in just before the lemma, then we obtain w k p ( k ; r,s ) v = − n − k k n − k k n − k
00 1 k ζ n − k
00 0 0 1 k v ( n − k ) v ( k ) = − n − k k n − k k v ( n − k ) v ( k ) ζ v ( n − k ) = − ζ v ( n − k ) v ( k ) v ( n − k ) , which proves the lemma. (cid:3) Theorem 2.7.
Let B n ⊂ L be a Borel subgroup of L . A double flag variety G/P S × L/B n has finitely many L -orbits. In other words, G/P S has finitely many B n -orbits. In thissense, G/P S is a real L -spherical variety.Proof. Firstly, we consider the open Bruhat cell, i.e., the case where k = 0 and w = w = J n . The cell is isomorphic to w − P S w / ( w − P S w ∩ P S ) ≃ N w , where N w = n(cid:16) n z n (cid:17) (cid:12)(cid:12)(cid:12) z ∈ Sym n ( R ) o (2.18)and the action of h ∈ GL n ( R ) ≃ L is given by the unimodular action: z hz t h . Sothe complete representatives of L -orbits are given by { z r,s := diag(1 r , − s , | r, s ≥ EAL DOUBLE FLAG VARIETIES 11 , r + s ≤ n } . Let H r,s ⊂ L be the stabilizer of z r,s (note that H r,s is denote as H (0; r,s ) inLemma 2.5. We omit 0 for brevity). Then an L -orbit in the open Bruhat cell is isomorphicto L/H r,s . What we must prove is that there are only finitely many B n orbits on L/H r,s .Direct calculations tell that H r,s = n(cid:16) α β γ (cid:17) ∈ GL n ( R ) (cid:12)(cid:12)(cid:12) α ∈ O( r, s ) , γ ∈ GL n − ( r + s ) ( R ) o ⊂ GL n ( R ) , (2.19)where O( r, s ) denotes the indefinite orthogonal group preserving a quadratic form definedby diag(1 r , − s ). Note that if r + s = n , we simply get H r,s = O( r, s ), which is asymmetric subgroup in GL n ( R ). It is well known that a minimal parabolic subgroup P min has finitely many orbits on G/H , where G is a general connected reductive Lie group,and H its symmetric subgroup (i.e., an open subgroup of the fixed point subgroup of anon-trivial involution of G ). For this, we refer the readers to [Wol74], [Mat79], [Ros79].Thus, the Borel subgroup B n has an open orbit on L/ O( r, s ) when r + s = n . This isequivalent to say that b ′ n + o ( r, s ) = l for some choice b ′ n of a Borel subalgebra of l = Lie L .On the other hand, the following is known. Lemma 2.8.
Let G be a connected reductive Lie group and P min its minimal parabolicsubgroup. For any closed subgroup H of G , let us consider an action of H on the flagvariety G/P min by the left translation. Then the followings are equivalent. (1)
There are finitely many H -orbits in G/P min , i.e., we have H \ G/P min < ∞ . (2) There exists an open H -orbit in G/P min . (3) There exists g ∈ G for which Ad g · h + p min = g holds. For the proof of this lemma, see [KO13, Remark 2.5 4)]. There is a misprint there,however. So we repeat the remark here. Matsuki [Mat91] observed that the lemmafollows if it is valid for real rank one case, while the real rank one case had been alreadyestablished by Kimelfeld [Kim87]. See also [KS13] for another proof.Now, in the case where r + s < n , since the upper left corner of H r,s is O( r, s ), we canfind a Borel subgroup b ′ n in l for which b ′ n + h r,s = l holds. By the above lemma, H r,s hasfinitely many orbits in L/B n or B n \ L/H r,s < ∞ .Secondly, let us consider the general Bruhat cell. Then, by Lemma 2.5, we know thereare finitely many L -orbits and they are isomorphic to L/H ( k ; r,s ) . The Lie algebra of H ( k ; r,s ) realized in GL n ( R ) is of the following form: h ( k ; r,s ) = n α β γ δ η (cid:12)(cid:12)(cid:12) α ∈ o ( r, s ) o ⊂ gl n ( R ) , (2.20)where (cid:16) α β γ (cid:17) ∈ gl n − k ( R ) and δ ∈ M k,n − k ( R ) , η ∈ gl k ( R ). Let us choose a Borelsubalgebra b ′ n − k of gl n − k ( R ) such that b ′ n − k + { (cid:16) α β γ (cid:17) | α ∈ o ( r, s ) } = gl n − k ( R ) , applying the arguments for the open Bruhat cell. Then we can take b n = (cid:18) b ′ n − k ∗ b k (cid:19) as a Borel subalgebra of l which satisfies b n + h ( k ; r,s ) = l . Thus Lemma 2.8 tells that B n -orbits in L/H ( k ; r,s ) is finite. (cid:3) Corollary 2.9.
For any parabolic subgroup Q of L , the double flag variety X = Λ × Ξ d = G/P S × L/Q has finitely many L -orbits, hence it is of finite type. Maslov index
In [KS94], Kashiwara and Shapira described the orbit decomposition of the diagonalaction of G = Sp n ( R ) in the triple product Λ = Λ × Λ × Λ of Lagrangian Grassmannians.They used an invariant called Maslov index to classify the orbits and concluded that thereare only finitely many orbits, i.e., /G < ∞ .Let us explain the relation of their result and ours.Fix points x ± ∈ Λ which are corresponding to the Lagrangian subspaces V ± ⊂ V . Weconsider a G -stable subspace containing { x + } × { x − } × Λ, namely Put Y = G · (cid:16) { x + } × { x − } × Λ (cid:17) . Since all the orbits go through a point { x + } × { x − } × { λ } for a certain λ ∈ Λ, G -orbitdecomposition of Y reduces to orbit decomposition of Stab G ( { x + } × { x − } ) in Λ = G/P S .It is easy to see that the stabilizer Stab G ( { x + } × { x − } ) is exactly L so that Y /G ≃ Λ /L ≃ L \ G/P S , on the last of which we discussed in §
2. Since Y ⊂ Λ , it has finitelymany G -orbits due to [KS94], hence Λ = G/P S also has finitely many L -orbits. A detailedlook at [KS94] will also provides the classification of orbits, which we do not carry outhere.However, for proving the finiteness of B n -orbits, we need explicit structure of orbits ashomogeneous spaces of L . This is the main point of our analysis in § Classification of open L -orbits in the double flag variety Let us return back to the original situation of Grassmannians, i.e., our Q = P GL( d,n − d ) ⊂ L is a maximal parabolic subgroup which stabilizes a d -dimensional subspace in V + . So thedouble flag variety X = G/P S × L/Q is isomorphic to the product of the LagrangianGrassmannian Λ = LGr( R n ) and the Grassmannian Ξ d = Gr d ( R n ) of d -dimensionalsubspaces.In this section, we will describe open L -orbits in X . To study L -orbits in X = G/P S × L/Q , we use the identification
X/L ≃ Q \ G/P S ≃ Λ /Q. In this identification, open L -orbits corresponds to open Q -orbits, since they are of thelargest dimension. We already know the description of L -orbits on Λ = G/P S from § Q -orbits are necessarily contained in open L -orbits, hence we concentrate on the EAL DOUBLE FLAG VARIETIES 13 open Bruhat cell P S w P S /P S ≃ N w ≃ Sym n ( R ). L acts on Sym n ( R ) via unimodularaction: h · z = hz t h ( z ∈ Sym n ( R ) , h ∈ GL n ( R ) ≃ L ).The following lemma, Sylvester’s law of inertia, is a special case of Lemma 2.3. Lemma 4.1.
Let L = GL n ( R ) act on N w = Sym n ( R ) via unimodular action. Then,open orbits are parametrized by the signature ( p, q ) with p + q = n . A complete system ofrepresentatives are given by { I p,q | p, q ≥ , p + q = n } , where I p,q = diag(1 p , − q ) . Let us denote open L -orbits byΩ( p, q ) = { z ∈ Sym n ( R ) | z has signature ( p, q ) } = { hI p,q t h | h ∈ GL n ( R ) } . (4.1)Thus we are looking for open Q -orbits in Ω( p, q ). Let us denote H = Stab L ( I p,q ), thestabilizer of I p,q ∈ Ω( p, q ), which is isomorphic to an indefinite orthogonal group O( p, q ).As a consequence Ω( p, q ) ≃ L/H ≃ GL n ( R ) / O( p, q ).Since Ω( p, q ) ≃ L/H , Ω( p, q ) /Q ≃ H \ L/Q ≃ Ξ d /H, where Ξ d = Gr d ( R n ) is the Grassmannian of d -dimensional subspaces. So our problem ofseeking Q -orbits in Ω( p, q ) is equivalent to understand H -orbits in a partial flag varietyΞ d . Since H is a symmetric subgroup fixed by an involutive automorphism of L , thisproblem is ubiquitous in representation theory of real reductive Lie groups.Let us consider a d -dimensional subspace U = span R { e , e , . . . , e d } ∈ Gr d ( R n ) which isstabilized by Q . Take z ∈ Ω( p, q ), and consider a quadratic form Q z − ( v, v ) = t v z − v ( v ∈ R n ) associated to z − , which also has the same signature ( p, q ) as that of z . Note thatthe restriction of Q z − to U can be degenerate, and the rank and the signature of Q z − (cid:12)(cid:12) U is preserved by the action of Q . In fact, for u ∈ U and m ∈ Q , we get Q ( m · z ) − ( u, u ) = t u ( mz t m ) − u = t u ( t m − z − m − ) u = t ( m − u ) z − ( m − u ) = Q z − ( m − u, m − u ) . Since m − ∈ Q preserves U , the quadratic forms Q z − and Q ( m · z ) − have the same rankand the signature when restricted to U . So they are clearly invariants of a Q -orbit inΩ( p, q ). Put Ω( p, q ; s, t ) = { z ∈ Ω( p, q ) | Q z − (cid:12)(cid:12) U has signature ( s, t ) } , (4.2)where s + t is the rank of Q z − (cid:12)(cid:12) U . Clearly 0 ≤ s ≤ p, ≤ t ≤ q and s + t ≤ d must besatisfied. Lemma 4.2. Q -orbits in Ω( p, q ) are exactly { Ω( p, q ; s, t ) | s, t ≥ , t + p ≥ d, s + q ≥ d, s + t ≤ d } given in (4.2) . The orbit Ω( p, q ; s, t ) is open if and only if s + t = d = dim U , i.e., thequadratic form Q z − is non-degenerate when restricted to U . Proof.
The restriction Q z − (cid:12)(cid:12) U is a quadratic form, and we denote its signature by ( s, t ).The rank of Q z − (cid:12)(cid:12) U is s + t and k = d − ( s + t ) is the dimension of the kernel. Obviously,we must have 0 ≤ s, t, k ≤ d . Since Q z − is non-degenerate with signature ( p, q ), thereexist signature constraints s + k ≤ p, t + k ≤ q. These conditions are equivalent to the condition given in the lemma. The signature ( s, t )and hence the dimension k of the kernel is invariant under the action of Q .Conversely, if a d -dimensional subspace U of the quadratic space R n has the samesignature ( s, t ) (and hence k ), it can be translated into U by the isometry group O( p, q )by Witt’s theorem. This means the signature concretely classifies Q -orbits. (cid:3) This lemma practically classifies open L -orbits on X = Λ × Ξ d . However, we rewrite itmore intrinsically.Firstly, we note that, for z ∈ Sym n ( R ), a Lagrangian subspace λ ∈ Λ =
G/P S in theopen Bruhat cell P S w P S /P S ≃ N w is given by λ = { v = (cid:16) zxx (cid:17) | x ∈ R n } , and clearly such z is uniquely determined by λ . We denote the Lagrangian subspace by λ z . Also, we denote a d -dimensional subspace in Ξ d = Gr d ( R n ) by ξ . Theorem 4.3.
Suppose that non-negative integers p, q and s, t satisfies p + q = n, s + t = d, ≤ s ≤ p, ≤ t ≤ q. (4.3) Then an open L -orbit in X = Λ × Ξ d is given by O ( p, q ; s, t ) = { ( λ z , ξ ) ∈ Λ × Ξ d | sign( z ) = ( p, q ) , sign( Q z − (cid:12)(cid:12) ξ ) = ( s, t ) } . Every open orbit is of this form. Relative invariants
Let us consider the vector space Sym n ( R ) × M n,d ( R ), on which GL n ( R ) × GL d ( R ) acts.The action is given explicitly as( h, m ) · ( z, y ) = ( hz t h , hy t m )(( h, m ) ∈ GL n ( R ) × GL d ( R ) , ( z, y ) ∈ Sym n ( R ) × M n,d ( R )) . Let us put M ◦ n,d ( R ) := { y ∈ M n,d ( R ) | rank y = d } , the subset of full rank matri-ces in M n,d ( R ). Then, a map π : M ◦ n,d ( R ) → Ξ d = Gr d ( R n ) defined by π ( y ) :=span R { y j | ≤ j ≤ d } ( y j denotes the j -th column vector of y ) is a quotient map by EAL DOUBLE FLAG VARIETIES 15 the action of GL d ( R ). Thus we get a diagram:Sym n ( R ) × M n,d ( R ) o o open ? _ Sym n ( R ) × M ◦ n,d ( R ) / GL d ( R ) (cid:15) (cid:15) Sym n ( R ) × Ξ d (cid:31) (cid:127) open / / Λ × Ξ d Comparing to the Grassmannian, the vector space Sym n ( R ) × M n,d ( R ) is easier to handle.In particular, we introduce two basic relative invariants ψ and ψ on ( z, y ) ∈ Sym n ( R ) × M n,d ( R ) with respect to the above linear action, ψ ( z, y ) = det z, ψ ( z, y ) = det z · det( t y z − y ) . Note that ψ ( z, y ) = ( − d det (cid:18) z y t y (cid:19) , so that it is actually a polynomial. We consider two characters of ( h, m ) ∈ GL n ( R ) × GL d ( R ): χ ( h, m ) = (det h ) , χ ( h, m ) = (det h ) (det m ) . (5.1)Then it is easy to check that the relative invariants ψ , ψ are transformed under characters χ − , χ − respectively. Let us define e Ω = { ( z, y ) ∈ Sym n ( R ) × M n,d ( R ) | ψ ( z, y ) = 0 , ψ ( z, y ) = 0 } , e Ω( p, q ; s, t ) = { ( z, y ) ∈ Sym n ( R ) × M n,d ( R ) | sign( z ) = ( p, q ) , sign( t y z − y ) = ( s, t ) } . The set e Ω is clearly open and is a union of open GL n ( R ) × GL d ( R )-orbits in Sym n ( R ) × M n,d ( R ). Theorem 5.1.
The sets e Ω( p, q ; s, t ) , where p + q = n, s + t = d, ≤ s ≤ p, ≤ t ≤ q, (5.2) are open GL n ( R ) × GL d ( R ) -orbits, and they exhaust all the open orbits in Sym n ( R ) × M n,d ( R ) , i.e., e Ω = a p,q,s,t e Ω( p, q ; s, t ) , where the union is taken over p, q, s, t which satisfies (5.2) . Moreover, the quotient e Ω( p, q ; s, t ) / GL d ( R ) is isomorphic to Ω( p, q ; s, t ) , an open L -orbit in the double flag variety X = Λ × Ξ d . This theorem is just a paraphrase of Theorem 4.3.Since relative invariants are polynomials, we can consider them on the complexifiedvector space Sym n ( C ) × M n,d ( C ). In the rest of this section, we will study them on thiscomplexified vector space, and we denote it simply by Sym n × M n,d omitting the base field.Similarly, we use GL n = GL n ( C ), etc., for algebraic groups over C . Recall the characters χ , χ of GL n × GL d in (5.1). The following theorem should bewell-known to the experts, but we need the proof of it to get further results. Theorem 5.2. GL n × GL d -module Pol(Sym n × M n,d ) contains a unique non-zero relativeinvariant f ( z, y ) with character χ − m χ − m ( m , m ≥ up to non-zero scalar multiple.This relative invariant is explicitly given by f ( z, y ) = (det z ) m + m (det( t y z − y )) m .Proof. In this proof, to avoid notational complexity, we consider the dual action( h, m ) · ( z, y ) = ( t h − zh − , t h − ym − ) (( z, y ) ∈ Sym n × M n,d , ( h, m ) ∈ GL n × GL d ) . To translate the results here to the original action is easy.First, we quote results on the structure of the polynomial rings over Sym n and M n,d .Let us denote the irreducible finite dimensional representation of GL n with highest weight λ by V ( n ) ( λ ) (if n is to be well understood, we will simply write it as V ( λ )). Lemma 5.3. (1)
As a GL n -module, Sym n is multiplicity free, and the irreducible de-composition of the polynomial ring is given by Pol(Sym n ) ≃ M µ ∈P n V ( n ) (2 µ ) . (5.3)(2) Assume that n ≥ d ≥ . As a GL n × GL d -module, M n,d is also multiplicity free, andthe irreducible decomposition of the polynomial ring is given by Pol(M n,d ) ≃ M λ ∈P d V ( n ) ( λ ) ⊗ V ( d ) ( λ ) . (5.4)Since we are looking for relative invariants for GL d , it must belong to one dimensionalrepresentation space det ℓd = V ( d ) ( ℓ̟ d ), where ̟ d = (1 , . . . , , , . . . ,
0) denotes the d -thfundamental weight. Thus it must be contained in the space (cid:0) V ( n ) (2 µ ) ⊗ V ( n ) ( ℓ̟ d ) (cid:1) ⊗ V ( d ) ( ℓ̟ d ) ⊂ Pol(Sym n × M n,d ) . (5.5)Since a relative invariant is also contained in the one dimensional representation of GL n ,say det kn = V ( n ) ( k̟ n ), V ( n ) (2 µ ) ⊗ V ( n ) ( ℓ̟ d ) must contain V ( n ) ( k̟ n ). We argue V ( n ) (2 µ ) ⊗ V ( n ) ( ℓ̟ d ) ⊃ V ( n ) ( k̟ n ) ⇐⇒ µ − k̟ n = ( ℓ̟ d ) ∗ = ℓ̟ n − d − ℓ̟ n ⇐⇒ µ = ( k − ℓ ) ̟ n + ℓ̟ n − d . Thus, ℓ ≥ k − ℓ ≥ µ . So therelative invariant is unique (up to a scalar multiple) if we fix the character det nk det dℓ = χ ( k − ℓ ) / χ ℓ/ . (cid:3) Corollary 5.4.
Let us consider the relative invariant f ( z, y ) = (det z ) m + m (det( t y z − y )) m ( m , m ≥ in the above theorem. EAL DOUBLE FLAG VARIETIES 17 (1)
The space span C { f ( z, y ) | z ∈ Sym n } ⊂ Pol(M n,d ) is stable under GL n and it isisomorphic to V ( n ) (2 m ̟ d ) ∗ ⊗ V ( d ) (2 m ̟ d ) ∗ as a GL n × GL d -module. (2) Similarly, the space span C { f ( z, y ) | y ∈ M n,d } ⊂ Pol(Sym n ) is stable under GL n andit is isomorphic to V ( n ) (2 m ̟ n + 2 m ̟ n − d ) ∗ .Proof. It is proved that f ( z, y ) ∈ (cid:0) V ( n ) (2 µ ) ⊗ V ( n ) ( ℓ̟ d ) (cid:1) ⊗ V ( d ) ( ℓ̟ d ) ⊂ Pol(Sym n × M n,d ) , where k = 2 m + 2 m , ℓ = 2 m and µ = ( k − ℓ ) ̟ n + ℓ̟ n − d . For any specializationof y , this space is mapped to V ( n ) (2 µ ) (or possibly zero), and if we specialize z to somesymmetric matrix, it is mapped to V ( n ) ( ℓ̟ d ) ⊗ V ( d ) ( ℓ̟ d ). This shows the results. (cid:3) Although, we do not need the following lemma below, it will be helpful to know theexplicit formula for det( t y z − y ). Note that we take z instead of z − in the lemma. Lemma 5.5.
Let [ n ] = { , , . . . , n } and put (cid:0) [ n ] d (cid:1) := { I ⊂ [ n ] | I = d } , the familyof subsets in [ n ] of d -elements. For X ∈ M n and I, J ∈ (cid:0) [ n ] d (cid:1) , we will denote X I,J :=( x i,j ) i ∈ I,j ∈ J , a d × d -submatrix of X . For ( z, y ) ∈ Sym n × M n,d , we have det( t y zy ) = X I,J ∈ ( [ n ] d ) det( z IJ ) det(( y t y ) IJ ) = X I,J ∈ ( [ n ] d ) det( z IJ ) det( y I, [ d ] ) det( y J, [ d ] ) . We observe that { det( y I, [ d ] ) | I ∈ (cid:0) [ n ] d (cid:1) } is the Pl¨ucker coordinates and also { det( z IJ ) | I, J ∈ (cid:0) [ n ] d (cid:1) } is the coordinates for the determinantal variety of rank d (there are muchabundance though).6. Degenerate principal series representations
Let us return back to the situation over real numbers, and we introduce degenerateprincipal series for G = Sp n ( R ) and L = GL n ( R ) respectively.6.1. Degenerate principal series for
G/P S . Let us recall G = Sp n ( R ) and its maximalparabolic subgroup P S . Take a character χ P S of P S , and consider a degenerate principalseries representation C ∞ - Ind GP S χ P S := { f : G → C : C ∞ | f ( gp ) = χ P S ( p ) − f ( g ) ( g ∈ G, p ∈ P S ) } , where G acts by left translations: π Gν ( g ) f ( x ) = f ( g − x ). In the following, we will take χ P S ( p ) = | det a | ν for p = (cid:16) a w t a − (cid:17) ∈ P S . (6.1)(We can multiply the sign sgn(det a ) by χ P S , if we prefer.)Since Sym n ( R ) is openly embedded into G/P S , a function f ∈ C ∞ - Ind GP S χ P S is de-termined by the restriction f (cid:12)(cid:12) Ω where Ω is the embedded image of Sym n ( R ) in G/P S .Explicitly, Ω is defined byΩ = (cid:26) J (cid:16) z (cid:17) P S /P S | z ∈ Sym n ( R ) (cid:27) , J = (cid:16) − n n (cid:17) = w , where w is the longest element in the Weyl group, and we give an open embedding bySym n ( R ) / / Ω (cid:31) (cid:127) open / / G/P S z ✤ / / J (cid:16) z (cid:17) P S /P S (6.2)In the following we mainly identify Sym n ( R ) and Ω. Let us give the fractional linearaction of G on Sym n ( R ) in our setting. Lemma 6.1.
In the above identification, the linear fractional action g . z of g = (cid:16) a bc d (cid:17) ∈ G on z ∈ Sym n ( R ) = Ω is given by g . z = − ( az − b )( cz − d ) − ∈ Sym n ( R ) , (6.3) if det( cz − d ) = 0 .Proof. By the identification, w = g . z corresponds to gJ (cid:16) z (cid:17) P S /P S . We can calculateit as gJ (cid:16) z (cid:17) = J ( J − gJ ) (cid:16) z (cid:17) = J (cid:16) d − c − b a (cid:17)(cid:16) z (cid:17) = J (cid:16) d − cz − c − b + az a (cid:17) = J (cid:16) w (cid:17)(cid:16) d − cz − c u (cid:17) , where w = ( az − b )( d − cz ) − and u = a + wc = t ( d − cz ) − . This proves the desired formula. (cid:3)
Lemma 6.2.
For f ∈ C ∞ - Ind GP S χ P S , the action of π Gν ( g ) on f is given by π Gν ( g ) f ( z ) = | det( a + zc ) | − ν f ( g − . z ) ( g = (cid:16) a bc d (cid:17) ∈ G, z ∈ Sym n ( R )) , where χ P S ( p ) is given in (6.1) . In particular, for h = (cid:16) a t a − (cid:17) ∈ L , we get π Gν ( h ) f ( z ) = | det( a ) | − ν f ( a − z t a − ) . We want to discuss the completion of the C ∞ -version of the degenerate principal series C ∞ - Ind GP S χ P S to a representation on a Hilbert space. Usually, this is achieved by thecompact picture, but here we use noncompact picture. To do so, we need an elementarydecomposition theorem.Here we write P S = LN S , L = M A , where we wrote N for N S which is the unipotentradical of P S , and M = SL ± n ( R ) , A = R + . Further, we denote M K = M ∩ K = O( n ). EAL DOUBLE FLAG VARIETIES 19
For the opposite Siegel parabolic subgroup P S , we denote a Langlands decomposition by P S = M AN S .Thus we conclude N S M AN S ⊂ KM AN S = G (open embedding). Every g ∈ G can bewritten as g = kman ∈ KM AN S , and we call this generalized Iwasawa decomposition byabuse of the terminology. Iwasawa decomposition g = kman may not be unique, but ifwe require ma = (cid:16) h t h − (cid:17) for an h ∈ Sym + n ( R ), it is indeed unique. This follows fromthe facts that the decomposition M = O( n ) · Sym + n ( R ) is unique (Cartan decomposition),and that M K = K ∩ M = O( n ).Now we describe an explicit Iwasawa decomposition of elements in N S . Lemma 6.3.
Let v ( z ) := (cid:16) z (cid:17) ∈ N S ( z ∈ Sym n ( R )) and denote h := √ n + z ∈ Sym + n ( R ) , a positive definite symmetric matrix. Then we have the Iwasawa decomposition v ( z ) = kman ∈ KM AN = G, where k = h − (cid:16) − zz (cid:17) = (cid:16) h − − h − zh − z h − (cid:17) , h = p n + z ,ma = (cid:16) h t h − (cid:17) , n = (cid:16) t h − zh − (cid:17) a = α n , α = (det(1 + z )) n Proof.
Since (cid:16) z − z (cid:17)(cid:16) z (cid:17) = (cid:16) z z (cid:17) , we get (putting h = √ z )1 √ z (cid:16) z − z (cid:17)(cid:16) z (cid:17) = (cid:16) h h − z h − (cid:17) , = (cid:16) h t h − (cid:17)(cid:16) t h − zh − (cid:17) . Notice that 1 √ z (cid:16) z − z (cid:17) is in K , and its inverse is given by k in the statement ofthe lemma. The rest of the statements are easy to derive. (cid:3) Since N S M AN S is open dense in G , f ∈ C ∞ - Ind GP S χ P S is determined by f (cid:12)(cid:12) N S . Wecomplete the space of functions on N S or Sym n ( R ) by the measure (det(1 + z )) ν − n +12 dz ,where ν = Re ν and dz denotes the usual Lebesgue measure, in order to get a Hilbertrepresentation. See [Kna86, § VII.1] for details (we use unnormalized induction, so thatthere is a shift of ρ P S ( a ) = | det(Ad( a ) (cid:12)(cid:12) N S ) | / = | det a | n +12 ). Thus our Hilbert space is H Gν := { f : Sym n ( R ) → C | k f k G,ν < ∞} , where k f k G,ν := Z Sym n ( R ) | f ( z ) | (det(1 + z )) ν − n +12 dz. (6.4)We denote an induced representation Ind GP S χ P S on the Hilbert space H Gν by π Gν . Remark 6.4. The degenerate principal series Ind GP S χ P S induced from the character χ P S ( p ) = | det a | ν (cf. Eq. (6.1)) has the unitary axis at ν = n +12 . If n is even, thereexist complementary series for real ν which satisfies n < ν < n + 1 (see [Lee96, Th. 4.3]).6.2. Degenerate principal series for
L/Q . In this subsection, we fix the notationsfor degenerate principal series of L = GL n ( R ) from its maximal parabolic subgroup Q = P GL( d,n − d ) . We will denote q = (cid:16) k q k ′ (cid:17) ∈ Q, and χ Q ( q ) = | det k | µ . (6.5)Then χ Q is a character of Q , and we consider a degenerate principal series representation C ∞ - Ind LQ χ Q := { F : L → C : C ∞ | F ( aq ) = χ Q ( q ) − F ( a ) ( a ∈ L, q ∈ Q ) } , where L acts by left translations: π Lµ ( a ) F ( Y ) = f ( a − Y ) ( a, Y ∈ L ). We introduce an L -norm on this space just like usual integral over a maximal compact subgroup K L = K ∩ L = O( n ): k F k L,µ := Z K L | F ( k ) | dk ( F ∈ C ∞ - Ind LQ χ Q ) , (6.6)and take a completion with respect to this norm to get a Hilbert space H Lµ . Note thatthe integration is in fact well-defined on K L / ( K ∩ Q ) ≃ O( n ) / O( d ) × O( n − d ), becauseof the right equivariance of F . Thus we get a representation π Lµ = Ind LQ χ Q on the Hilbertspace H Lµ .To make the definition of intertwiners more easy to handle, we unfold the Grassmannian L/Q ≃ Gr d ( R n ). Recall M ◦ n,d ( R ) = { y ∈ M n,d ( R ) | rank y = d } . Then, we get a map L = GL n ( R ) / / M ◦ n,d ( R ) Y = (cid:16) y y y y (cid:17) ✤ / / y = (cid:16) y y (cid:17) (6.7)which induces an isomorphism Ξ d = L/Q ∼ −→ M ◦ n,d ( R ) / GL d ( R ). Thus we can identify C ∞ - Ind LQ χ Q with the space of C ∞ functions F : M ◦ n,d ( R ) → C with the property F ( yk ) = | det k | − µ F ( y ). In this picture, the action of L is just the left translation: π Lµ ( a ) F ( y ) = F ( a − y ) ( y ∈ M ◦ n,d ( R ) , a ∈ GL n ( R ) = L ) . To have the L -norm defined in (6.6), we restrict the projection map (6.7) from L = GL n to K L = O( n ), the resulting space being the Stiefel manifold of orthonormal frames S n,d = { y ∈ M ◦ n,d | t y y = 1 d } . Then L/Q is isomorphic to S n,d / O( d ). The norm given in(6.6) is equal to k F k L,µ = Z S n,d | F ( v ) | dσ ( v ) , where dσ ( v ) is the uniquely determined O( n )-invariant non-zero measure. Note that S n,d ≃ O( n ) / O( n − d ). EAL DOUBLE FLAG VARIETIES 21
Remark 6.5. The degenerate principal series Ind LQ χ Q induced from the character χ Q ( q ) = | det k | µ (cf. Eq. (6.5)) is never unitary as a representation of GL n ( R ). However, if werestrict it to SL n ( R ), it has the unitary axis at µ = n − d . In addition, there existcomplementary series for real µ in the interval of n − d − < µ < n − d + 1 (see [HL99, § y part to 1 d . Thuswe get ay = (cid:16) a a a a (cid:17)(cid:16) y (cid:17) = (cid:16) a + a y a + a y (cid:17) = (cid:16) a + a y )( a + a y ) − (cid:17) , and the fractional linear action is given by y ( a + a y )( a + a y ) − ( y ∈ M n − d,d ( R )) . Intertwiners between degenerate principal series representations
In this section, we consider the following kernel function K α,β ( z, y ) := | det( z ) | α | det( t y z − y ) | β = | det( z ) | α − β (cid:12)(cid:12) det (cid:16) z y t y (cid:17)(cid:12)(cid:12) β (( z, y ) ∈ Sym n ( R ) × M n,d ( R )) , (7.1)with complex parameters α, β ∈ C . Using this kernel, we aim at defining two integralkernel operators P and Q , which intertwine degenerate principal series representations.7.1. Kernel operator P from π Gν to π Lµ . In this subsection, we define an integral kerneloperator P for f ∈ C ∞ - Ind GP S χ P S with compact support in Ω( p, q ): P f ( y ) = Z Ω( p,q ) f ( z ) K α,β ( z, y ) dω ( z ) ( y ∈ M ◦ n,d ( R )) , (7.2)where dω ( z ) is an L -invariant measure on the open L -orbit Ω( p, q ) ⊂ Ω. So the operator P depends on the parameters p and q as well as α and β .For h = (cid:16) a t a − (cid:17) ∈ L and f above, we have P ( π Gν ( h ) f )( y ) = Z Ω( p,q ) χ P S ( a ) − f ( a − z t a − ) K α,β ( z, y ) dω ( z )= χ P S ( a ) − Z Ω( p,q ) f ( z ) K α,β ( az t a , y ) dω ( az t a )= χ P S ( a ) − Z Ω( p,q ) f ( z ) | det( a ) | α K α,β ( z, a − y ) dω ( z )= | det( a ) | α − ν Z Ω( p,q ) f ( z ) K α,β ( z, a − y ) dω ( z )= | det( a ) | α − ν π Lµ ( a ) P f ( y ) . Thus, if ν = 2 α , we get an intertwiner. In this case, we have P f ( yk ) = | det( k ) | β P f ( y )so that P f ( y ) ∈ C ∞ - Ind LQ χ Q for χ Q ( p ) = | det k | − β ( p = (cid:16) k ∗ k ′ (cid:17) ) , if it is a C ∞ -function on L/Q . To get an intertwiner to π Lµ , we should have 2 β = − µ .As we observed Λ = G/P S ⊃ [ p + q = n Ω( p, q ) (open) . For each p, q , the space H Lµ ( p, q ) := L (Ω( p, q ) , (det(1 + z )) ν − n +12 dz ) is a closed subspaceof H Gν and L -stable. From the decomposition of the base spaces, we get a direct sumdecomposition of L -modules: H Gν = M p + q = n H Gν ( p, q )Now we state one of the main theorem in this section. Theorem 7.1.
Let ν := Re ν, µ := Re µ and assume that they satisfy inequalities nν + dµ > n ( n + 1)2 , nν − dµ > n ( n + 1)2 , (7.3) and ν + µ ≥ n + 1 , µ ≤ . (7.4) Put α = ν/ , β = − µ/ . Then the integral kernel operator P f defined in (7.2) convergesand gives a bounded linear operator P : H Gν ( p, q ) → H Lµ which intertwines π Gν (cid:12)(cid:12) L to π Lµ . The rest of this subsection is devoted to prove the theorem above. Mostly we omit p, q if there is no misunderstandings and we write ν, µ instead of ν , µ in the following.Let us evaluate the square of integral |P f ( y ) | point wise. The first evaluation is givenby Cauchy-Schwartz inequality: |P f ( y ) | ≤ Z Ω | f ( z ) | (det(1 + z )) ν − n +12 dz Z Ω | K α,β ( z, y ) | (det(1 + z )) − ( ν − n +12 ) | det z | − ( n +1) dz ≤ k f k G,ν Z Ω | K α,β ( z, y ) | (det(1 + z )) − ( ν − n +12 ) | det z | − ( n +1) dz, where dz is the Lebesgue measure on Sym n ( R ) ≃ R n ( n +1)2 , and we use dω ( z ) = | det z | − n +12 dz . Since α = ν/ β = − µ/
2, the second integral becomes Z Ω | K α,β ( z, y ) | (det(1 + z )) − ( ν − n +12 ) | det z | − ( n +1) dz = Z Ω | det z | ν + µ − ( n +1) (cid:12)(cid:12) det (cid:16) z y t y (cid:17)(cid:12)(cid:12) − µ (det(1 + z )) − ( ν − n +12 ) dz. (7.5) EAL DOUBLE FLAG VARIETIES 23
To evaluate the last integral, we use polar coordinates for z . Namely, we put r := √ trace z and write z = r Θ. Then trace(Θ ) = 1, and Ω Θ ( p, q ) = Ω( p, q ) ∩ { Θ | trace(Θ ) = 1 } iscompact. Using polar coordinates, we get det z = r n det Θ anddet (cid:16) z y t y (cid:17) = det (cid:16) r Θ y t y (cid:17) = r − d det (cid:16) r Θ ry t ( ry ) 0 (cid:17) = r n − d det (cid:16) Θ y t y (cid:17) Also we note that dz = r n ( n +1)2 − drd Θ. Thus we get Z Ω( p,q ) | det z | ν + µ − ( n +1) (cid:12)(cid:12) det (cid:16) z y t y (cid:17)(cid:12)(cid:12) − µ (det(1 + z )) − ( ν − n +12 ) dz = Z Ω Θ ( p,q ) | det Θ | ν + µ − ( n +1) (cid:12)(cid:12) det (cid:16) Θ y t y (cid:17)(cid:12)(cid:12) − µ d Θ × Z ∞ r n ( ν + µ − ( n +1))+( n − d )( − µ ) (det(1 + r Θ )) − ( ν − n +12 ) r n ( n +1)2 − dr = Z Ω Θ ( p,q ) | det Θ | ν + µ − ( n +1) (cid:12)(cid:12) det (cid:16) Θ y t y (cid:17)(cid:12)(cid:12) − µ d Θ × Z ∞ r nν + dµ − n ( n +1)2 − (det(1 + r Θ )) − ( ν − n +12 ) dr By the assumption (7.4), the integrand in the first integral over Ω Θ ( p, q ) is continuous,and converges. For the second, we separate it according as r ↓ r ↑ ∞ .If r is near zero, the factor det(1 + r Θ ) is approximately 1, so the integral converges if R r nν + dµ − n ( n +1)2 − dr converges. The first inequality in (7.3) guarantees the convergence.On the other hand, if r is large, the factor det(1 + r Θ ) is asymptotically r n , sothe integral converges if R ∞ r nν + dµ − n ( n +1)2 − − n ( ν − n +12 ) dr converges. We use the secondinequality in (7.3) to conclude the convergence.Thus the integral (7.5) does converge, and the square root of it gives a bound for theoperator norm of P . We finished the proof of Theorem 7.1.7.2. Kernel operator Q from π Lµ to π Gν . Similarly, we define Q F ( z ), for the moment,for F ( y ) ∈ C ∞ - Ind LQ χ Q by Q F ( z ) = Z M n,d ( R ) F ( y ) K α,β ( z, y ) dy ( z ∈ Sym n ( R )) , (7.6)where dy denotes the Lebesgue measure on M n,d ( R ). We will update the definition of Q afterwards in (7.7), although we will check L -equivariance using this expression. The integral (7.6) may diverge , but at least we can formally calculate as( Q π Lµ ( a ) F )( z ) = Z M n,d ( R ) F ( a − y ) K α,β ( z, y ) dy = Z M n,d ( R ) F ( y ) K α,β ( z, ay ) daydy dy = Z M n,d ( R ) F ( y ) | det a | α K α,β ( a − z t a − , y ) | det a | d dy = | det a | α + d χ P S ( h ) π Gν ( h ) Q F ( z ) . Thus, if χ P S ( h ) − = | det a | α + d , we get an intertwiner. Here, we need a compatibility forthe action, i.e., F ( yk ) = | det k | − µ F ( y ) ( k ∈ GL d ( R )) and we get F ( yk ) K α,β ( z, yk ) d ( yk ) = | det k | − µ +2 β + n F ( y ) K α,β ( z, y ) dy. From this we can see, if µ = 2 β + n , the integrand (or measure) F ( y ) K α,β ( z, y ) dy is definedover M ◦ n,d ( R ) / GL d ( R ) ≃ O( n ) / O( d ) × O( n − d ). This last space is compact. Instead of thisfull quotient, we use the Stiefel manifold S n,d introduced in § n,d ( R ). Thus,for α = − ( ν + d ) / β = ( µ − n ) /
2, we redefine the intertwiner Q by Q F ( z ) = Z S n,d F ( y ) K α,β ( z, y ) dσ ( y ) ( z ∈ Sym n ( R )) , (7.7)where dσ ( y ) denotes the O( n )-invariant measure on S n,d . Theorem 7.2.
Let ν := Re ν, µ := Re µ and assume that they satisfy inequalities nν + dµ < n ( n + 1)2 , nν − dµ < n ( n + 1)2 , (7.8) and ν + µ ≤ n − d, µ ≥ n. (7.9) If α = − ( ν + d ) / and β = ( µ − n ) / , the integral kernel operator Q defined in (7.7) converges and gives an L -intertwiner Q : H Lµ → H Gν . Two remarks are in order. First, the inequalities (7.8) and (7.9) is “opposite” to theinequalities in Theorem 7.1. So ( ν, µ ) does not share a common region for convergence.Second, the condition (7.9) in fact implies (7.8). However, we suspect the inequality (7.9)is too strong to ensure the convergence. So we leave them as they are.Now let us prove the theorem. For brevity, we denote ν , µ by ν, µ in the following.Since α − β = − ( ν + d ) / − ( µ − n ) / ( n − d − ( ν + µ )) ≥ β = ( µ − n ) / ≥ K α,β ( z, y ) is continuous. So the integral (7.7) converges. Let us check Q F ( z ) ∈ H Gν for F ∈ H Lµ . By Cauchy-Schwarz inequality, we get |Q F ( z ) | ≤ Z S n,d | F ( y ) | dσ ( y ) Z S n,d | K α,β ( z, y ) | dσ ( y ) = k F k L,µ Z S n,d | K α,β ( z, y ) | dσ ( y ) . EAL DOUBLE FLAG VARIETIES 25
Thus kQ F k G,ν = Z Sym n ( R ) |Q F ( z ) | (det(1 + z )) ν − n +12 dz ≤ k F k L,µ Z S n,d Z Sym n ( R ) | K α,β ( z, y ) | (det(1 + z )) ν − n +12 dzdσ ( y )Since α = − ( ν + d ) / β = ( µ − n ) /
2, the integral of square of the kernel is Z Sym n ( R ) | K α,β ( z, y ) | (det(1 + z )) ν − n +12 dz = Z Sym n ( R ) | det z | − ( ν + µ )+ n − d (cid:12)(cid:12) det (cid:16) z y t y (cid:17)(cid:12)(cid:12) µ − n (det(1 + z )) ν − n +12 dz. (7.10)As in the proof of Theorem 7.1, we use polar coordinate z = r Θ. Namely, we put r := √ trace z and write z = r Θ. If we put Ω Θ = { Θ ∈ Sym n ( R ) | trace(Θ ) = 1 } , it iscompact and dz = r n ( n +1)2 − drd Θ. Thus we get Z Sym n ( R ) | det z | − ( ν + µ )+ n − d (cid:12)(cid:12) det (cid:16) z y t y (cid:17)(cid:12)(cid:12) µ − n (det(1 + z )) ν − n +12 dz = Z Ω Θ | det Θ | − ( ν + µ )+ n − d (cid:12)(cid:12) det (cid:16) Θ y t y (cid:17)(cid:12)(cid:12) µ − n d Θ × Z ∞ r n ( − ( ν + µ )+ n − d )+( n − d )( µ − n ) (det(1 + r Θ )) ν − n +12 r n ( n +1)2 − dr = Z Ω Θ | det Θ | − ( ν + µ )+ n − d (cid:12)(cid:12) det (cid:16) Θ y t y (cid:17)(cid:12)(cid:12) µ − n d Θ × Z ∞ r − ( nν + dµ )+ n ( n +1)2 − (det(1 + r Θ )) ν − n +12 dr. Since the integrand in the first integral over Ω Θ is continuous and hence converges. Forthe second, we separate it according as r ↓ r ↑ ∞ as in the proof of Theorem 7.1.When r is near zero, the integral converges if R r − ( nν + dµ )+ n ( n +1)2 − dr converges. Theconvergence follows from The first inequality in (7.8). When r is large, the integralconverges if R ∞ r − ( nν + dµ )+ n ( n +1)2 − n ( ν − n +12 ) dr converges. We use the second inequality in(7.8) for the convergence.This completes the proof of Theorem 7.2.7.3. Finite dimensional representations. If α, β ∈ Z , we can naturally consider analgebraic kernel function K α,β ( z, y ) = det( z ) α det( t y z − y ) β (( z, y ) ∈ Sym n ( R ) × M n,d ( R )) without taking absolute value. By abuse of notation, we use the same symbol as before.Similarly we also consider algebraic characters χ P S ( p ) = det( a ) ν ( p = (cid:16) a ∗ t a − (cid:17) ∈ P S )and χ Q ( q ) = det( k ) µ ( q = (cid:16) k ∗ k ′ (cid:17) ∈ Q )if µ and ν are integers. In this setting the results in the above subsections are also valid.We make use of Corollary 5.4 to deduce the facts on the image and kernels of integralkernel operators considered above. Theorem 7.3.
For nonnegative integers m and m , we put α = m + m , β = m anddefine K α,β ( z, y ) as above. (1) Put ν = − m + m ) − d and µ = 2 m + n , and define the characters χ P S and χ Q as above. Then Ind LQ χ Q contains the finite dimensional representation V ( n ) (2 m ̟ n +2 m ̟ n − d ) ∗ as an irreducible quotient. On the other hand, the representation Ind GP S χ P S contains the same finite dimensional representation of L as a subrepresentation, and Q intertwines these two representations. This subrepresentation is the same for any p and q . (2) Assume m ≥ n + 1 and put ν = 2( m + m ) and µ = − m . Define the charac-ters χ P S and χ Q as above. Then Ind GP S χ P S contains the finite dimensional representation V ( n ) (2 m ̟ d ) ∗ of L = GL n ( R ) as an irreducible quotient. On the other hand Ind LQ χ Q contains the same finite dimensional representation as a subrepresentation, and P inter-twines these two representations. The intertwiners depend on p and q , so there are atleast ( n + 1) different irreducible quotients which is isomorphic to V ( n ) (2 m ̟ d ) ∗ , whilethe image in Ind LQ χ Q is the same.Proof. This follows immediately from Corollary 5.4 and Theorems 7.1 and 7.2. Note that2 m ≥ n + 1 is required for the convergence of the integral operator. (cid:3) The above result illustrates how knowledge about the geometry of a double flag varietyand associated relative invariants may give information about the structure of paraboli-cally induced representations, and in particular about some branching laws. Let us ex-plain, that the branching laws in the above Theorem are consistent with other approachesto the structure of Ind GP S χ P S in Theorem 7.3 (1).Let us in the following remind about the connection between this induced representa-tion, living on the Shilov boundary S of the Hermitian symmetric space G/K , and thestructure of holomorphic line bundles on this symmetric space. Let g = k + p be a Cartandecomposition, and p C = p + ⊕ p − be a decomposition into irreducible representations of K .For holomorphic polynomials on the symmetric space we have the Schmid decomposition(see [FK94], XI.2.4) of the space of polynomials P ( p + ) = ⊕ a P a ( p + ) EAL DOUBLE FLAG VARIETIES 27 and the sum is over multi-indices a of integers with α ≥ α ≥ · · · ≥ α n ≥
0, labeling(strictly speaking, here one chooses an order so that these are the negative of) K -highestweights α γ + · · · + α n γ n with γ , . . . , γ n Harish-Chandra strongly orthogonal non-compactroots. Now by restricting polynomials to the Shilov boundary S we obtain an imbedding ofthe Harish-Chandra module corresponding to holomorphic sections of the line bundle withparameter ν in the parabolically induced representation on S with the same parameter.For concreteness, recall: For f ∈ C ∞ - Ind GP S χ P S , the action of π Gν ( g ) on f is given by π Gν ( g ) f ( z ) = | det( a + zc ) | − ν f ( g − . z ) ( g = (cid:16) a bc d (cid:17) ∈ G, z ∈ Sym n ( R )) . When ν is an even integer, this is exactly the action in the (trivialized) holomorphicbundle, now valid for holomorphic functions of z ∈ Sym n ( C ). So if we can find parameterswith a finite-dimensional invariant subspace in this Harish-Chandra module, then thesame module will be an invariant subspace in Ind GP S χ P S .Recall that the maximal compact subgroup K of G has a complexification isomorphicto that of L , and the G/K is a Hermitian symmetric space of tube type. Indeed, insidethe complexified group G C the two complexifications are conjugate. Hence if we considera finite-dimensional representation of G (or G C ), then the branching law for each of thesesubgroups will be isomorphic.For Hermitian symmetric spaces of tube type in general also recall the reproducingkernel (as in [FK94], especially Theorem XIII.2.4 and the notation there) for holomorphicsections of line bundles on G/K , h ( z, w ) − ν = X a ( ν ) a K a ( z, w ) (7.11)and the sum is again over multi-indices a of integers with α ≥ α ≥ · · · ≥ α n ≥ K a ( z, w ) are (suitably normalized) reproducing kernels of the K -representations P a ( p + ). The Pochhammer symbol is in terms of the scalar symbol in ourcase here( ν ) a = ( ν ) α ( ν − / α · · · ( ν − ( n − / α n = n Y i =1 ( ν − ( i − / α i , and( x ) k = x ( x + 1) · · · ( x + k −
1) = Γ( x + k )Γ( x ) . Recall that for positive-definiteness of the above kernel, ν must belong to the so-calledWallach set; this means that the corresponding Harish-Chandra module is unitary andcorresponds to a unitary reproducing-kernel representation of G (or a double covering of G ). Here the Wallach set is W = { , , . . . , n − } ∪ ( n − , ∞ )as in [FK94], XIII.2.7. On the other hand, if ν is a negative integer, the Pochhammer symbols ( ν ) a vanisheswhen α > − ν . So this gives a finite sum in the formula (7.11) for the reproducingkernel corresponding to a finite-dimensional representation of G , and a labels the K -types occurring here as precisely those with − ν ≥ α . By taking boundary values weobtain an imbedding of the K -finite holomorphic sections on G/K to sections of theline bundle on
G/P S . Recalling that for our G the Harish-Chandra strongly orthogonalnon-compact roots are 2 e j in terms of the usual basis e j , this means that the L -types inTheorem 7.3 (1) indeed occur. Namely, we may identify the parameters by the equation2 m ̟ n + 2 m ̟ n − d = 2( m + m , . . . , m + m , m , . . . , m )with the right-hand side of the form of a multi-index a satisfying − ν = 2( m + m )+ d ≥ α as required above.Thus we have seen, that there is consistency with the results about branching lawsfrom G to K coming from considering finite-dimensional continuations of holomorphicdiscrete series representations, and on the other hand those branching laws from G to L coming from our study of relative invariants and intertwining operators from Ind GP S χ P S to Ind LQ χ Q . References [BJ06] Armand Borel and Lizhen Ji,
Compactifications of symmetric and locally symmetric spaces ,Mathematics: Theory & Applications, Birkh¨auser Boston, Inc., Boston, MA, 2006.[BSKZ14] Salem Ben Said, Khalid Koufany, and Genkai Zhang,
Invariant trilinear forms on sphericalprincipal series of real rank one semisimple Lie groups , Internat. J. Math. (2014), no. 3,1450017, 35.[CKØP11] Jean-Louis Clerc, Toshiyuki Kobayashi, Bent Ørsted, and Michael Pevzner, GeneralizedBernstein-Reznikov integrals , Math. Ann. (2011), no. 2, 395–431.[FGT09] Michael Finkelberg, Victor Ginzburg, and Roman Travkin,
Mirabolic affine Grassmannianand character sheaves , Selecta Math. (N.S.) (2009), no. 3-4, 607–628.[FK94] Jacques Faraut and Adam Kor´anyi, Analysis on symmetric cones , Oxford MathematicalMonographs, The Clarendon Press, Oxford University Press, New York, 1994, Oxford SciencePublications.[FN16] Lucas Fresse and Kyo Nishiyama,
On the exotic Grassmannian and its nilpotent variety ,Represent. Theory (2016), 451–481, Paging previously given as: 1–31.[HL99] Roger Howe and Soo Teck Lee, Degenerate principal series representations of GL n ( C ) and GL n ( R ), J. Funct. Anal. (1999), no. 2, 244–309.[HNOO13] Xuhua He, Kyo Nishiyama, Hiroyuki Ochiai, and Yoshiki Oshima, On orbits in double flagvarieties for symmetric pairs , Transform. Groups (2013), no. 4, 1091–1136.[HT12] Anthony Henderson and Peter E. Trapa, The exotic Robinson-Schensted correspondence , J.Algebra (2012), 32–45.[Kim87] Boris Kimelfeld,
Homogeneous domains on flag manifolds , J. Math. Anal. Appl. (1987),no. 2, 506–588.[KK79] Masaki Kashiwara and Takahiro Kawai,
On holonomic systems for Π Nl =1 ( f l +( √− λ l , Publ.Res. Inst. Math. Sci. (1979), no. 2, 551–575.[KM14] T. Kobayashi and T. Matsuki, Classification of finite-multiplicity symmetric pairs , Transform.Groups (2014), no. 2, 457–493. EAL DOUBLE FLAG VARIETIES 29 [Kna86] Anthony W. Knapp,
Representation theory of semisimple groups , Princeton MathematicalSeries, vol. 36, Princeton University Press, Princeton, NJ, 1986, An overview based on exam-ples.[KO13] Toshiyuki Kobayashi and Toshio Oshima,
Finite multiplicity theorems for induction and re-striction , Adv. Math. (2013), 921–944.[Kob05] Toshiyuki Kobayashi,
Multiplicity-free representations and visible actions on complex mani-folds , Publ. Res. Inst. Math. Sci. (2005), no. 3, 497–549.[KØP11] Toshiyuki Kobayashi, Bent Ørsted, and Michael Pevzner, Geometric analysis on small unitaryrepresentations of
GL( N, R ), J. Funct. Anal. (2011), no. 6, 1682–1720.[KS94] Masaki Kashiwara and Pierre Schapira, Sheaves on manifolds , Grundlehren der Mathematis-chen Wissenschaften [Fundamental Principles of Mathematical Sciences], vol. 292, Springer-Verlag, Berlin, 1994, With a chapter in French by Christian Houzel, Corrected reprint of the1990 original.[KS13] B. Kroetz and H. Schlichtkrull,
Finite orbit decomposition of real flag manifolds , ArXiv e-prints (2013).[KS15] Toshiyuki Kobayashi and Birgit Speh,
Symmetry breaking for representations of rank oneorthogonal groups , Mem. Amer. Math. Soc. (2015), no. 1126, v+110.[Lee96] Soo Teck Lee,
Degenerate principal series representations of
Sp(2 n, R ), Compositio Math. (1996), no. 2, 123–151.[Mat79] Toshihiko Matsuki, The orbits of affine symmetric spaces under the action of minimal para-bolic subgroups , J. Math. Soc. Japan (1979), no. 2, 331–357.[Mat88] , Closure relations for orbits on affine symmetric spaces under the action of parabolicsubgroups. Intersections of associated orbits , Hiroshima Math. J. (1988), no. 1, 59–67.[Mat91] , Orbits on flag manifolds , Proceedings of the International Congress of Mathemati-cians, Vol. I, II (Kyoto, 1990), Math. Soc. Japan, Tokyo, 1991, pp. 807–813.[MØO16] Jan M¨ollers, Bent Ørsted, and Yoshiki Oshima,
Knapp-Stein type intertwining operators forsymmetric pairs , Adv. Math. (2016), 256–306.[NO11] Kyo Nishiyama and Hiroyuki Ochiai,
Double flag varieties for a symmetric pair and finitenessof orbits , J. Lie Theory (2011), no. 1, 79–99.[Ros79] W. Rossmann, The structure of semisimple symmetric spaces , Canad. J. Math. (1979),no. 1, 157–180.[Tra09] Roman Travkin, Mirabolic Robinson-Schensted-Knuth correspondence , Selecta Math. (N.S.) (2009), no. 3-4, 727–758.[Wol74] Joseph A. Wolf, Finiteness of orbit structure for real flag manifolds , Geometriae Dedicata (1974), 377–384.[Zha09] Genkai Zhang, Radon, cosine and sine transforms on Grassmannian manifolds , Int. Math.Res. Not. IMRN (2009), no. 10, 1743–1772.
Department of Physics and Mathematics, Aoyama Gakuin University, Fuchinobe 5-10-1,Sagamihara 252-5258, Japan
E-mail address : [email protected] Department of Mathematics, Aarhus University, Ny Munkegade, 8000 Aarhus C, Den-mark
E-mail address ::