Restriction for general linear groups: the local non-tempered Gan-Gross-Prasad conjecture (non-Archimedean case)
aa r X i v : . [ m a t h . R T ] J un RESTRICTION FOR GENERAL LINEAR GROUPS: THE LOCALNON-TEMPERED GAN-GROSS-PRASAD CONJECTURE(NON-ARCHIMEDEAN CASE)
KEI YUEN CHAN
Abstract.
We prove a local Gan-Gross-Prasad conjecture on predicting the branchinglaw for the non-tempered representations of general linear groups in the case of non-Archimedean field. We also generalize to Bessel and Fourier-Jacobi models and studya possible generalization to Ext-branching laws. Introduction
In 1990s’, Gross-Prasad [GP93] formulated conjectures which determine when an irre-ducible generic representation of SO n − ( F ) appears in a quotient of an irreducible genericrepresentation of SO n ( F ) , where F is a local field. The conjectural answer is in termsof symplectic root numbers, providing deep connections with number theory. About tenyears ago, Gan-Gross-Prasad [GGP12] generalized the conjectures to other classical groups.The local generic conjectures in orthogonal, unitary and symplectic-metaplectic cases havebeen respectively settled by Waldspurger [Wa12], Mœglin-Waldspurger [MW12], and byBeuzart-Plessis [BP14], Gan-Ichino [GI16], and by Atobe [At18].Recently, Gan-Gross-Prasad [GGP19] formulated new conjectures for certain nontem-pered representations arising from a local component of an automrophic representation.The main goal of this paper is to prove one of those conjectures for general linear groupsover a non-Archimedean local field and study related generalizations.1.1. Local non-tempered Gan-Gross-Prasad conjecture.
We begin with a preciseformulation of the non-tempered conjecture. Let G n = GL n ( F ) , the general linear groupover a local field F . Let Alg( G n ) be the category of smooth G n -representations.Let W F be the Weil group of F . The Weil-Deligne group W D F of F is defined as: W D F = (cid:26) W F × SL ( C ) if F is non-Archimedean W F if F is ArchimedeanThe set of Langlands parameters of G n is the set of equivalence classes of φ : W D F → L G = GL n ( C ) , under conjugation by elements in GL n ( C ) , and the restriction to the factor of SL ( C ) in W F is algebraic. The local Langlands correspondence for GL n ( F ) is now known by[HT01, He00, Sc13].Define the Arthur parameters [Ar89] as the set of L G -orbits of maps ψ : W D F × SL ( C ) → L G such that ψ | W D F has bounded image i.e. has tempered Langlands parameter, and therestriction to the SL ( C ) factor is algebraic. For each Arthur parameter ψ , one assigns a L -parameter given by φ ψ ( w ) = ψ ( w, (cid:18) | w | / | w | − / (cid:19) ) . Let
Sym k ( C ) be the unique ( k + 1) -dimensional representation of SL ( C ) . The Arthurparameter, as a finite W D F × SL ( C ) -representation ψ , takes the form M A = X d M d ⊗ Sym d ( C ) , (1.1)where each M d is a representation of W D F such that ψ | W D F has bounded image i.e. each M i corresponds to a tempered representation. It gives rise to a Langlands parameter M as described above, and gives a G n -representation denoted by π M . Any irreducible smoothrepresentation of G n associated to the Langlands parameter φ ψ coming from an Arthurparameter is called a representation of Arhtur type.A key notion in [GGP19] is the relevant pair which governs the branching law of repre-sentations of Arthur type: Definition 1.1. [GGP19] Two Arthur parameters M A and N A are said to form a rele-vant pair if there exists W D F -representations M +0 , . . . , M + r , M − , . . . , M − s correspondingto tempered representations such that M A = r X d =0 M + d ⊗ Sym d ( C ) ⊕ s X d =1 M − d ⊗ Sym d − ( C ) , (1.2)and N A = r X d =1 M + d ⊗ Sym d − ( C ) ⊕ s X d =0 M − d ⊗ Sym d ( C ) . (1.3)We regard G n as a subgroup of G n +1 via the embedding g diag( g, . A non-temperedGan-Gross-Prasad conjecture predicts which Arthur type representations of G n appears inthe quotient of an Arthur type representation of G n +1 , in terms of relevant pairs. Conjecture 1.2. [GGP19, Conjecture 5.1]
Let F be a local field. Let π M and π N be Arthurtype representations of GL n +1 ( F ) and GL n ( F ) respectively. Then Hom G n ( π M , π N ) = 0 ifand only if their respective associated Arthur parameters M A and N A are relevant. The main result of the paper is to prove the conjecture for non-Archimedean field F .Previously, for non-Archimedean F , certain cases including when the Deligne SL ( C ) in W D F acts trivially are proved in [GGP19], and the only if direction is proved by M. Gure-vich [Gu18]. We shall give another proof for the only if direction in this paper. Recently,Gourevitch-Sayag [GS20] have results towards the Archimedean case. Theorem 1.3. If F is non-Archimedean, Conjecture 1.2 holds. ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 3
Representation-theoretic reformulation.
From now on, we assume F is non-Archimedean. We first reformulate the problem into a representation theory setup.For the detailed notions of Zelevinsky segments and product, see Section 2.3. For anirreducible unitarizable cuspidal representation ρ of G l , let ∆ ρ ( m ) = [ ν − ( m − / ρ, ν ( m − / ρ ] be a Zelevinsky segment. Any square integrable representation is known to be isomorphic to St(∆ ρ ( m )) for some such Zelevinsky segment ∆ ρ ( m ) [Ze80]. Any tempered representationis isomorphic to a product of some square-integrable representations, and corresponds to a W D F -representation ψ with bounded image ψ ( W D F ) .Let v ρ ( m, d ) be the unique irreducible quotient of the product St( ν ( d − / ∆ ρ ( m )) × . . . × St( ν − ( d − / ∆ ρ ( m )) , which is so-called a Speh representation and is unitarizable. Each factor M d ⊗ Sym d ( C ) in(1.1) corresponds to a product of Speh representations of the form v ρ ( m , d ) × . . . × v ρ r ( m r , d ) . (1.4)Any Arthur type representation is a product of some Speh representations. It follows from[Be84, Ta86] that such product is irreducible, and is independent of the ordering of Spehrepresentations.For an irreducible π ∈ Alg( G r ) , let e π be the highest derivative of π and let π − = ν / e π ,where ν ( g ) = | det g | F . A key observation in [GGP19] is that v ρ ( m, d ) − ∼ = v ρ ( m, d − , (1.5)and so ( v ρ ( m , d ) × . . . × v ρ r ( m r , d )) − ∼ = v ρ ( m , d − × . . . × v ρ r ( m r , d − , (1.6)which is also a motivation for the notion of relevant pairs in [GGP19]. The isomorphism(1.5) follows from the well-known highest derivative of Zelevinsky [Ze80] (and its translationto the Zelevinsky classification via [Ta86]).Thus combining Definition 1.1, (1.4) and (1.6), we have the following reformulation: Reformulation of Conjecture 1.2 for non-Archimedean.
Let F be a non-Archimedeanlocal field. Let π M and π N be Arthur type representations of GL n +1 ( F ) and GL n ( F ) re-spectively. Then Hom G n ( π M , π N ) = 0 if and only if there exist Speh representations π p, , . . . , π p,r and π q, , . . . , π q,s such that π M ∼ = π p, × . . . × π p,r × π − q, × . . . × π − q,s and π N ∼ = π − p, × . . . × π − p,r × π q, × . . . × π q,s . KEI YUEN CHAN
Generalizations.
The first generalization is on Bessel and Fourier-Jacobi models(Theorem 5.11). Such generalization is expected in [GGP19], but exact formulation wasnot stated. The strategy for proving general cases is connecting those models via Bernstein-Zelevinsky theory and then using the reduction to basic case as in [GGP12]. The notion ofBessel models and Fourier-Jacobi models in this paper is slightly general than the one in[GGP12, Section 15].In more detail, let H Rr = g x v t u : g ∈ G r , x ∈ M at r × ( n − r ) , v ∈ F n − r , u ∈ U n − r ⊂ G n +1 , where U n − r is the subgroup of unipotent upper triangular matrices. It is sometimes referredto a Rankin-Selberg subgroup. Let ψ be a generic character on the subgroup U n − r ⋉ F n − r ,extending trivially to H Rr . We show that the restriction problem for a Bessel model or aFourier-Jacobi model is equivalent to the problem of determining the corresponding Rankin-Selberg model (Corollary 6.4), i.e. determining if Hom H Rr ( π ⊗ ψ ⊗ ν − ( n − r ) / , π ) = 0 , where π and π are respective irreducible G n +1 and G r representations.The second generalization is on Ext-branching laws. The generic case for Ext-branchinglaw is simpler: for respective generic irreducible representations π and π of G n +1 and G n , Hom G n ( π , π ) ∼ = C , and Ext iG n ( π , π ) = 0 , for i ≥ . The Ext-vanishing part is conjectured by D. Prasad [Pr18] and proved in [CS18b]. One mayconsider an analogous problem of Ext-branching laws for Arthur representations. However,there is no such general Ext-vanishing result for Arthur representations, and we do nothave a way predicting non-vanishing Ext at the moment.Nevertheless, we formulate a conjecture in Section 7.1, which reduces computations ofExt-groups for branching laws to computation of Ext-groups of derivatives. The conjectureis partly based on the derivative approach in [GGP19], as well as some examples computedin this paper.1.4.
Outline of the proof of non-tempered GGP.
We shall consider the reformulatedproblem in Section 1.2. Let π M = π p, × . . . × π p,r ∈ Alg( G n +1 ) , (1.7)and π N = π q, × . . . × π q,s ∈ Alg( G n ) , (1.8)where each π p,i and π q,j is an (irreducible) Speh representation.The proof is on the induction of the total number of factors π p,i and π q,j which are notcuspidal representations. The basic case is that all factors are cuspidal representations.Then the associated Arthur parameters M A and N A are automatically relevant. Since therepresentations π M and π N are generic in this case, we always have Hom G n ( π M , π N ) = 0 . ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 5
The strategy of the general case is to find a suitable filtration on π M | G n → λ → π M | G n → ω → such that Hom G n ( ω, π N ) = Ext G n ( ω, π N ) = 0 (1.9)and Hom G n ( λ, π N ) can be transferred to another Hom space computable from the inductivecase. Now a long exact sequence argument gives Hom G n ( π M | G n , π N ) ∼ = Hom G n ( λ, π N ) and so one concludes the former from the latter one. The way to find such filtration isbased on a combination of Bernstein-Zelevinsky filtration and Mackey theory, and (1.9)would follow from comparing cuspidal supports on ω and π N .In more detail, an Arthur type representation π M is written as a product of Speh rep-resentations in (1.7). Write π ′ = π p, × . . . × π p,r . Mackey theory gives a short exactsequence: → π p, | M × π ′ → ( π p, × π ′ ) | M → π p, × ( π ′ | M ) → , where | M is the restriction to the mirabolic subgroup (for the detailed notations on theproduct involving mirabolic subgroups, see Section 3). One further restricts from themirabolic subgroup M n +1 to G n to obtain a short exact sequence for G n -representations.Now a significant property of an Arthur type representation is that the Speh representa-tions in the product of (1.7) commute, and most of time (we shall explain more on this inthe last paragraph), one can choose a particular form so that one can show, via comparingcuspidal supports, Ext iG n (( π p, × ( π ′ | M )) | G n , π N ) = 0 (1.10)and, from Bernstein-Zelevinsky theory, π p, | M admits a short exact sequence, as mirabolicsubgroup representations, sequence: → ( ν − / π − p, ) × Π → π p, | M → Q → , where Π is the Gelfand-Graev representation and Q is a quotient, such that for all i , againby comparing cuspidal supports, Ext iG n (( Q × π ′ ) | G n , π N ) = 0 (1.11)Thus standard homological algebra transfers the study of Hom G n ( π M , π N ) to Hom G n ((( ν − / π − p, ) × Π × π ′ ) | G n , π N ) Thus it remains to study the last Hom or study the structure of (( ν − / π − p, ) × Π × π ′ ) | G n .The later one is equivalent to study π − p, × ((Π × π ′ ) | G k ) (1.12)for some k , for which we will deduce information from the representation π − p, × (( σ × π ′ ) | G k ) (1.13) KEI YUEN CHAN for a suitable choice of unitarizable cuspidal representation σ . Now σ × π ′ is still an Arthurtype representation and so one can deduce information of the quotient of ( σ × π ′ ) | G k frominduction hypothesis.It is clear that if λ is a quotient of ( σ × π ′ ) | G k , then π − p, × λ is still a quotient of π − p, × (( σ × π ′ ) | G k ) , which basically deals with the if direction. The converse of the statementis not true in general, but holds under suitable assumption that fulfills our purpose. Forwhich, we have to study the product with π − p, preserves extensions in some situations(Corollary 9.4), which handles the only if direction.We finally explain the issue of choosing π p, to obtain the vanishing in (1.10) and (1.11).The choice does not always (easily) exist if we only consider the original restriction problemi.e. when Hom G n ( π M , π N ) = 0 . However, such choice exists either in the original restrictionproblem or in the dual restriction problem, which we mean the problem of determining if Hom G n +1 ( σ × ( π N ) ∨ | G n +1 , ( π M ) ∨ ) = 0 for a suitable choice of cuspidal representation σ of GL ( F ) . These two problems are indeedequivalent by Proposition 4.1. Such duality simplifies most of computations to comparingcuspidal supports.1.5. Remarks.
For irreducible generic quotients of G n appearing in an irreducible genericrepresentation of G n +1 (also known as generic GGP conjecture for GL -case), it is shown byRankin-Selberg integrals [JPSS83, Pr93]. In [CS18b], G. Savin and the author give anotherproof for the generic case using variations of Bernstein-Zelevinsky filtrations.Our method for Arthur type representations is again a variation of Bernstein-Zelevinskyfiltration method which exploits the product structure of Arthur representations. To illus-trate how the refinement gives more information, we consider respective representations in GL ( F ) and GL ( F ) in [GGP19, Remark 5.6]: π = h [ ν − , ν ] i × × , and π = h [ ν − / , ν / ] i × St([ ν − / , ν / ]) . (Here is the trivial character of F × .) Now the Mackey theory gives two layers on π | G : h [ ν − / , ν / ] i × ((1 × | G ) , and h [ ν − / , ν / ] i × ((1 | M × × | G ) Set τ = h [ ν − / , ν / ] i . A key difference of our method from the one in [GGP19] is to usetransfer in (1.12) and (1.13) to deduce that τ × ((1 | M × × | G ) has a quotient of π ,as G representations, which could deal with some obstruction we are going to describe inmore details.Now, in comparison with the full Bernstein-Zelevinsky filtration, the layer τ × ((1 | M × × | G ) further decomposes into three following layers: (for the notions of functors Ψ + , Φ + , seeSection 2.1) τ × ν / × ν / (multiplicity 1) , τ × (Φ + Ψ + (1) | G ) (multiplicity 2) ,τ × ((Φ + ) Ψ + (1) | G ) (multiplicity 1) . ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 7 (Here, the in the third layer is as G -representation.) Indeed, τ × ((Φ + ) Ψ + (1) | G ) contributes the non-zero Hom for Hom G ( π , π ) . However, in this decomposition, welose information on the extensions between these three layers, and give an obstruction onconcluding the non-vanishing Hom.An important ingredient in the transfer (such as from (1.12) to (1.13)) is that an ir-reducible cuspidal representation of G n restricted to the mirabolic subgroup is isomor-phic to the Gelfand-Graev representation, and such result is generalized to essentiallysquare-integrable representations when restricted to G n − via Hecke algebra realization[CS18b, CS19] (also see [Ch19] for further generalization to representations restricted tobe projective). Such fact also plays important roles in the reductions in [GGP12] and inproving the Ext-vanishing theorem in [CS18b].The only if direction is proved by M. Gurevich [Gu18], in which the Bernstein-Zelevinskyfiltration approach leads to an analysis of multiple products of generalized Speh represen-tations (which is called quasi-Speh representation in [Gu18]), arisen from the derivatives ofArthur type representations.Due to some transfer steps above, our approach only requires a study of producting withone Speh representations (rather than several generalized Speh representations), and weshow under some conditions on cuspidal supports that producting with a Speh represen-tation preserves extensions and is a fully-faithful functor. This improves one of results ofLapid-Mínguez [LM16] which shows producting with Speh representations preserves irre-ducibility under a related condition.1.6. Acknowledgement.
This project grows out from discussions with Dipendra Prasad,and the author would like to thank him for helpful discussions and comments. He wouldalso like to thank Gordan Savin for discussions on various topics and helpful comments.2.
Notations and Preliminaries
Bernstein-Zelevinsky functors.
For a connected reductive group G , let Alg( G ) bethe category of smooth representations of G . Let G n = GL n ( F ) . All representations inthis paper are smooth and we usually drop the term ’smooth’. For a representation π of G n , set n π = n .Let G = G n . For a closed subgroup H of G and a representation π in Alg( H ) , let Ind GH π to be the space of smooth functions f : G → π satisfying f ( hg ) = δ ( h ) / h.f ( g ) , where δ − is the modular character of H . The G -action on Ind GH π is given by ( g.f )( g ) = f ( g g ) forany g, g ∈ G . Let ind GH π be the subrepresentation of Ind GH π containing all functions withcompact support modulo H . We shall use u ind and u Ind for corresponding unnormalizedinductions of ind and
Ind respectively. Those functors
Ind , ind , u Ind , u Ind are exact [BZ76,Proposition 2.25(a)].Let M n be the mirabolic subgroup of G n i.e. M n is the subgroup of G n with all thematrices with the last row (0 , . . . , , . We shall also regard G n − as a subgroup of M n KEI YUEN CHAN via the embedding g (cid:18) g (cid:19) . Thus we have a chain of subgroups: G = M ⊂ . . . ⊂ M n − ⊂ G n − ⊂ M n ⊂ G n . For π ∈ Alg( G n ) , we may simply write π | M for the restriction π | M n Let V = V n be the unipotent radical of M n . Let ¯ ψ : F → C be a non-degeneratecharacter. Let ψ : V n → C by ψ ( v ) = ¯ ψ ( v n ) , where v n is the last entry in v . Note theaction of M n − stabilizes ψ : V n → C . For a character λ of V n and a representation π of M n , define π V n ,λ = δ − / π/ h v.x − λ ( v ) x : v ∈ V n , x ∈ π i , where δ − is the modular character of M n . When λ = 1 (resp. λ = ψ ), we regard as G n − -representation (resp. M n − -representation).Define θ = θ n : G n → G n by θ ( g ) = g − t , the Gelfand-Kazhdan involution [BZ76, Section7]. Define Φ + : Alg( M n ) → Alg( M n +1 ); Ψ + : Alg( G n ) → Alg( M n +1 )Φ − : Alg( M n +1 ) → Alg( M n ); Ψ − : Alg( M n +1 ) → Alg( G n ) . by Φ + ( π ) = ind M n +1 M n V n π ⊠ ψ, Ψ + ( π ) = ind M n +1 G n V n π ⊠ , Φ − ( π ) = π V n ,ψ , Ψ − ( π ) = π V n , . Some major properties of the functors [BZ77, Proposition 3.2]:(1) All the above functors are exact.(2) Φ − is left-adjoint to Φ + and Ψ − is left-adjoint to Φ + .(3) Φ − Ψ + = 0 and Ψ − Φ + = 0 (4) There is an exact sequence: → Φ + Φ − → Id → Ψ + Ψ − → (5) All the irreducible representations of M n are isomorphic to (Φ + ) k − Ψ + ( π ) for some k and some irreducible smooth G n − k -representation.(6) [BZ76, 5.18] For any cuspidal representation σ of G n , σ | M n ∼ = (Φ + ) n − (1) . Here is the -dimensional representation of M .Denote, the Gelfand-Graev representation, Π n := (Φ + ) n − (1) ∈ Alg( M n ) . (2.14)Let ν = ν n : G n → C be a character given by ν ( g ) = | det( g ) | F . For π ∈ Alg( G n ) , the k -th right and left derivatives of π are respectively defined as: π ( k ) = Ψ − (Φ − ) k − ( π | M n ) , ( k ) π = θ ( θ ( π ) ( k ) ) . and the k -th shifted right and left derivatives of π is defined as: π [ k ] = ν / · π ( k ) , [ k ] π = ν − / · ( k ) π. Let k ∗ be the largest integer such that π ( k ∗ ) = 0 . We shall call π ( k ∗ ) to be the highestderivative of π , and k ∗ is the level of π . We also set π − = π [ k ∗ ] . ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 9
Parabolic induction and Jacquet functors.
Let U n be the subgroup of G n con-taining all unipotent upper triangular matrices. Let N i be the unipotent subgroup of G n containing matrices of the form (cid:18) I n − i uI i (cid:19) for any ( n − i ) × n matrices u over F . Weregard G n − i × G i as a subgroup of G n via the embedding ( g , g ) diag( g , g ) . Let P i be the parabolic subgroup ( G n − i × G i ) N i .For π ∈ Alg( G n − i ) and π ∈ Alg( G i ) , define the product of π and π as π × π = Ind G n ( G n − i × G i ) ⋉ N i π ⊠ π ⊠ . For a family of representations π i ∈ Alg( G n i ) ( i = 1 , . . . , k ), define π × . . . × π k := π × ( . . . × ( π k − × π k ) . . . ) . The parabolic induction is an exact functor [BZ76]. For more properties for parabolicinductions, see [LM16].Let N − i = N ti be the opposite unipotent subgroup. For π ∈ Alg( G n ) , we shall de-note by π N i and π N − i be the corresponding normalized Jacquet modules, as G n − i × G i -representations. They are also exact functors. Since the parabolic induction has usual andopposite Jacquet functors as left and right adjoint functors respectively, parabolic inductionalso preserves injective and projective objects.For an irreducible representation π of G k , there is a unique set (with multiplicities) ofcuspidal representations ρ , . . . , ρ r such that π is a composition factor of ρ × . . . × ρ r , andwe denote the multiset cupp( π ) = { ρ , . . . , ρ r } , and denote cupp Z ( π ) = (cid:8) ν i ρ j (cid:9) i ∈ Z ,j =1 ,...,r .2.3. Speh representations and Zelevinsky segments.
Let ρ be an irreducible cuspidalrepresentation of G m . For any a, b ∈ C with b − a ∈ Z ≥ , a Zelevinsky segment ∆ =[ ν a ρ, ν b ρ ] is the set (cid:8) ν a ρ, ν a +1 ρ, . . . , ν b ρ (cid:9) , and we denote a (∆) = ν a ρ and b (∆) = ν b ρ .Denote by h ∆ i (resp. St(∆) ) the unique submodule (resp. quotient) of ν a ρ × . . . × ν b ρ .A Zelevinsky multisegment is a multiset of Zelevinsky segments. For a Zelevinsky mul-tisegment m = { ∆ , . . . , ∆ r } , denote by h m i the unique irreducible subrepresentation of h ∆ i × . . . × h ∆ r i , and denote by St( m ) the unique irreducible quotient of St(∆ ) × . . . × St(∆ r ) , where ∆ , . . . , ∆ r are ordered in the way as in [Ze80, Theorem 6.1]. We also denotethe parabolic induction h ∆ i × . . . × h ∆ r i by ζ ( m ) .Let ρ be an irreducible unitarizable cuspidal representation of G m . For a positive integer d , define ∆ ρ ( d ) = [ ν − ( d − / ρ, ν ( d − / ρ ] . For a positive integer m , define u ρ ( m, d ) = h n ν − ( m − / ∆ ρ ( d ) , . . . , ν ( m − / ∆ ρ ( d ) o i . We shall call those representations to be Speh representations, and they are unitarizable[Be84, Section 8] (see [Ta86]).In Section 1.2, we also introduce the notion v ρ ( m, d ) . The two notions coincide: Lemma 2.1. [Ta86, Theorem A10]
For any irreducible unitarizable cuspidal representation ρ , any d, m , v ρ ( m, d ) ∼ = u ρ ( m, d ) . Explicit derivatives of a Speh representation are particularly simple to describe, and onerefers to [LM14] (also see [CS19, Section 7]). We collect some useful information for ourstudy:
Lemma 2.2. [LM14, Theorem 14]
Let π = u ρ ( m, d ) be a Speh representation. (1) The level of π is n ρ m . (2) If k is not the level of π and π [ k ] = 0 , then the cuspidal support of π [ k ] contains ν ( d + m − / / ρ . (3) If k is the level of π , then π − = π [ k ] ∼ = u ρ ( m, d − and π ( k ) ∼ = ν − / u ρ ( m, d − . Mirabolic Induction
In this section, we discuss inductions involving mirabolic subgroups, which will be usedin Sections 4, 5 and 6.3.1.
Mirabolic induction.
Let τ ∈ Alg( M m ) and let π ∈ Alg( G n ) . Define two types ofmirabolic inductions, similar to [BZ77, 4.12].(1) Type 1: Let Q = P m ∩ M n + m ⊂ G n + m i.e. Q = (cid:26)(cid:18) g um (cid:19) : g ∈ G n , m ∈ M m , u ∈ M at n × m (cid:27) . Let ǫ : Q → C be the identity.(2) Type 2: Let Q = P tm ∩ M n + m ⊂ G n + m i.e. Q = gu h v : g ∈ G n , u ∈ M at m − ,n , h ∈ G m − , v ∈ F m − . Let ǫ : Q : → C given by ǫ = ν − / .For type 1 (resp. type 2), extend π ⊠ τ trivially to Q . Define the M n + m -representation π ¯ × τ (resp. τ ¯ × π ) to be the space of smooth functions f : M n + m → π ⊠ τ satisfying f ( qg ) = ǫ ( q ) δ ( q ) / q.f ( g ) for any q ∈ Q and g ∈ M n + m , and f is compactly-supportedmodulo Q , where δ − is the modular character of Q .In type 1, when restricting to G n + m − , we have ( π ¯ × τ ) | G n + m − ∼ = ( ν / π ) × ( τ | G m − ) , (3.15)where the isomorphism is given by f ( g f (diag( g, . Here we naturally identify π ⊠ τ and ( ν / π ) ⊠ ( τ | G m − ) . We may also sometimes simply write × for ¯ × . ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 11
Associative property.Lemma 3.1.
Let π ∈ Alg( G n ) . Let π ∈ Alg( G n ) . Let τ ∈ Alg( M r ) . Then (1) ( π ¯ × τ ) ¯ × π ∼ = π ¯ × ( τ ¯ × π ) ; and (2) ( π × π ) ¯ × τ ∼ = π ¯ × ( π ¯ × τ ) ; (3) ( τ ¯ × π ) ¯ × π ∼ = τ ¯ × ( π × π ) .Proof. Using induction in stages, the respective subgroups which are parabolically inducedfrom in LHS of (1), (2), (3) take the form: g ∗ g ∗ ∗∗ m ∗ , g ∗ ∗ ∗ g ∗ ∗ m ∗ , g ∗ g ∗ ∗ m ∗ and which are parabolically induced from RHS of (1), (2), (3) take the form: g ∗ ∗ ∗ g ∗ m ∗ , g ∗ ∗ ∗ g ∗ ∗ m ∗ , g ∗ g ∗ ∗ m ∗ , where g ∈ G n , g ∈ G n and m ∈ G r − .Let w = diag( (cid:18) I n I n (cid:19) , I r ) ∈ G n + n + r . Then w defines a G -map from ( π ¯ × τ ) ¯ × π to π ¯ × ( τ ¯ × π ) given by f f ( wg ) . This verifies (1). The same element defines a G -mapfor (3). (2) is easy. (cid:3) From parabolic to mirabolic induction.
The appearance of mirabolic inductionscomes from the study of parabolic inductions when restricting to the mirabolic subgroupvia Mackey theory. The following lemma will be used several times.
Lemma 3.2. [BZ77, Proposition 4.13]
Let π and π be G n and G n -representations.Then ( π × π ) | M admits a short exact sequence: → π | M ¯ × π → ( π × π ) | M → π ¯ × ( π | M ) → Connection to Bernstein-Zelevinsky functors.Lemma 3.3. [BZ77, Proposition 4.13]
Let π ∈ Alg( G n ) . Let τ ∈ Alg( M k ) . Then (1) Ψ − ( τ ¯ × π ) ∼ = Ψ − ( τ ) × π (2) → Φ − ( τ ) ¯ × π → Φ − ( τ ¯ × π ) → Ψ − ( τ ) ¯ × ( π | M ) → The following result is standard. We omit the details.
Lemma 3.4.
For π ∈ Alg( G r ) , (Φ + ) k Ψ + ( π ) ∼ = π ¯ × Π k +1 It is also convenient to define another functor:
Λ : Alg( G n ) → Alg( M n +1 ) by Λ( π ) = u Ind M n +1 G n ν − / π. By definitions, Λ( π ) ∼ = 1 | M ¯ × π . When n = 0 , Λ is just an isomorphism of vector spaces. Proposition 3.5.
Let r ≥ . Let π ∈ Alg( G r ) . For s ≥ , Π s +1 ¯ × π ∼ = (Φ + ) s (Λ( π )) . Proof.
For ≤ k ≤ s , by Lemma 3.3(2), (Φ − ) k (Π s +1 ¯ × π ) ∼ = Π s +1 − k ¯ × π , and, by Lemma3.3(1), (Ψ − ) k (Π s +1 ¯ × π ) = 0 . Thus from Bernstein-Zelevinsky theory [BZ77, Proposition3.2], we have Π s +1 ¯ × π ∼ = (Φ + ) s (Λ( π )) . (cid:3) A transfer lemma.
We shall need the following transfer or reduction:
Lemma 3.6.
Let π ∈ Alg( G k ) and π ∈ Alg( G l ) . Let π ∈ Alg( G n ) with n ≥ l + k . Let a = n + 1 − ( k + l ) . Then, for any irreducible cuspidal representation σ in Alg( G n +1 − ( k + l ) ) such that σ / ∈ csupp Z ( ν − / π ) , and for any i , Ext iG n ( π × (( σ × π ) | G n − k ) , π ) ∼ = Ext iG n ( π × ((Π a ¯ × π ) | G n − k ) , π ) . Proof.
Again Lemma 3.2 gives a filtration on ( σ × π ) | M n +1 − k as: → σ | M ¯ × π → ( σ × π ) | M → σ ¯ × ( π | M ) → . (3.16)Restricting to G n − k , this gives the filtration: → ( σ | M ¯ × π ) | G n − k → ( σ × π ) | G n − k → ( ν / σ ) × ( π | G l − ) → . With Π a = σ | M , producting with π gives the exact sequence: → π × ((Π a ¯ × π ) | G n − k ) → π × (( σ × π ) | G n − k ) → π × ( ν / σ ) × ( π | G l − ) → . (3.17)Thus standard argument using second adjointness of Frobenius reciprocity and compar-ing cuspidal support at ν / σ gives that, for all i , Ext iG n ( π × ( ν / σ ) × ( π | G l − ) , π ) = 0 . Thus long exact sequence from (3.17) gives that, for all i , Ext iG n ( π × ((Π a ¯ × π ) | G n − k ) , π ) ∼ = Ext iG n ( π × (( σ × π ) | G n − k ) , π ) . (cid:3) A lemma on Speh representation.Lemma 3.7.
Let π = u ρ ( m, d ) be a Speh representation. Let π ′ be in Alg( G k ) . Let n + 1 = n π + k . Let π ′′ be an irreducible representation of G n such that ν / ( ν ( m + d − / ρ ) is not in cupp( π ′′ ) . Then there exists a short exact sequence, as G n -representations: → K → ( π | M ¯ × π ′ ) | G n → Q → such that, for all i , Ext iG n ( Q, π ′′ ) = 0 , ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 13 and, K ∼ = (( ν − / u ρ ( m, d − × (Π p ¯ × π ′ )) | G n ∼ = u ρ ( m, d − × ((Π p ¯ × π ′ ) | G k + p − ) where p = n ρ m , and Ext iG n ( K, π ′′ ) ∼ = Ext iG n (( π | M ¯ × π ′ ) | G n , π ′′ ) . Proof.
From the bottom piece of Bernstein-Zelevinsky filtration (Lemma 2.2), π | M has thesubmodule (see Section 2.1 and Lemma 3.4) K ′ := ν − / u ρ ( m, d −
1) ¯ × Π p and ( π | M ) /K ′ admits a M -filtration whose successive quotients isomorphic to π ( j ) ¯ × Π j for j < p . Let G = G n . Now taking mirabolic is exact and so one would have, by a long exactsequence argument, Ext iG (( π | M ¯ × π ′ ) | G , π ′′ ) ∼ = Ext iG (( K ′ ¯ × π ′ ) | G , π ′′ ) if we can show that, for all i , Ext iG ((( π | M ) /K ′ ¯ × π ′ | G , π ′′ ) = 0 To show the last Ext vanishing, it suffices to show that for each piece of Bernstein-Zelevinsky layer τ = π ( j ) × Π j ( j < p ) appearing in ( π | M ) /K ′ , Ext iG (( τ ¯ × π ′ ) | G , π ′′ ) = 0 for any i , which indeed follows from: Ext iG ((( π ( j ) ¯ × Π j ) ¯ × π ′ ) | G , π ′′ ) ∼ =Ext iG (( ν / π ( j ) ) × ((Π j ¯ × π ′ ) | G j + k − ) , π ′′ ) ∼ =Ext iG nπ − j × G j + k − (( ν / π ( j ) ) ⊠ (Π j ¯ × π ′ ) , ( π ′′ ) N − j + k − ) ∼ =0 , where the first isomorphism follows from Lemma 3.1(1) and (3.15), the second isomorphismfollows from Frobenius reciprocity, and the last isomorphism follows from Lemma 2.2(2)with comparing cuspidal supports. (cid:3) Proof of Conjecture 1.2 (non-Archimedean)
The main goal of this section is to prove Conjecture 1.2 (non-Archimedean) moduloProposition 4.1 and Proposition 4.24.1.
Dual restriction.Proposition 4.1.
Let π and π be irreducible representations of G n +1 and G n respectively.For any irreducible cuspidal representation σ of G such that σ is not in cupp Z ( ν − / π ∨ ) ∪ cupp Z ( π ) , and for all i , Ext iG n ( π | G n , π ∨ ) ∼ = Ext iG n +1 (( π × σ ) | G n +1 , π ∨ ) . The proof of Lemma 4.1 will be postponed to Proposition 5.4, where we will prove amore general statement.
Product preserving quotients.Proposition 4.2.
Let ρ be an irreducible unitarizable cuspidal representation. Fix m, d .Let π be a (not necessarily admissible) representation of G n . Let p = n ρ md . Let π bean irreducible representation of G n + p such that any cuspidal representation in cupp( π ) iseither (1) lying in (cid:8) ν − ( m + d − / ρ, . . . , ν ( m + d − / ρ (cid:9) ; or (2) not lying in { ν n ρ } n ∈ Z .Then if Hom G n + p ( u ρ ( m, d ) × π , π ) = 0 , then there exists a non-zero irreducible quotient ω of π such that π ∼ = u ρ ( m, d ) × ω , more-over, if π is an irreducible Arthur type representation, then such ω is also an irreducibleArthur type representation. Proposition 4.2 will be proved as a special case of Corollary 9.4. Proposition 4.2 is onlyneeded for the only if direction.4.3.
Proof of non-tempered GGP.
Let S uk be the set of irreducible unitarizable cuspidalrepresentations of G k .The following two lemmas are the keys for reductions to an inductive case. Lemma 4.3.
Let π p and π q be Arthur type representations of G n +1 and G n respectively.Write π p = π p, × . . . × π p,r , π q = π q, × . . . × π q,s for some Speh representations π p,i , π q,j . Write π p,i = u ρ i ( m i , d i ) and π q,j = u σ j ( l j , e j ) .Suppose m + d ≥ m i + d i and m + d ≥ l j + e j for all i, j . Then Hom G n ( π p , π q ) = 0 if and only if for any e σ ∈ S un ρ m such that e σ / ∈ cupp Z ( π p ) ∪ cupp Z ( ν − / π q ) , Hom G n ( u ρ ( m , d − × (( e σ × π ′ p ) | G a ) , π q ) = 0 , where π ′ p = π p, × . . . × π p,r and a = n − n ρ m ( d − .Proof. By Lemma 3.2, → π p, | M ¯ × π ′ p → π p | M → π p, ¯ × ( π ′ p | M ) → . (4.18)Let n = n ρ d m and n ′ = n − n . Now Ext iG n (( π p, ¯ × ( π ′ p | M )) | G n , π q ) ∼ = Ext iG n (( ν / π p, ) × ( π ′ p | G n ′ ) , π q ) ∼ = Ext iG n × G n − n (( ν / π p, ) ⊠ ( π ′ p | G n ′ ) , ( π q ) N − n − n )= 0 , where the first isomorphism follows from (3.15) and Lemma 3.1(1) and the second isomor-phism follows from second adjointness of Frobenius reciprocity and the third isomorphismfollows by comparing cuspidal support at ν / ν ( d + m − / ρ . ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 15
Thus long exact sequence argument on (4.18) gives that, for all i , Ext iG n (( π p, | M ¯ × π ′ p ) | G n , π q ) ∼ = Ext iG n ( π p | G n , π q ) . (4.19)Set u ′ = π − p, ∼ = u ρ ( m , d − and u ′′ = ν − / u ′ . Now Lemma 3.7 gives that Ext iG n ((( u ′′ ¯ × Π) × π ′ p ) | G n , π q ) ∼ = Ext iG n (( π p, | M ¯ × π ′ p ) | G n , π q ) , (4.20)where Π = Π n ρ m .For any e σ ∈ S un ρ m not appearing in cupp Z ( π p ) ∪ cupp Z ( ν − / π q ) , Ext iG n ( u ′ × (( e σ × π ′ p ) | G t ) , π q ) ∼ = Ext iG n ( u ′ × ((Π ¯ × π ′ p ) | G t ) , π q ) ∼ = Ext iG n ((( u ′′ ¯ × Π) × π ′ p ) | G n , π q ) , (4.21)where t = n ′ + n ρ m . Here the first isomorphism follows from Lemma 3.6 and the secondisomorphism follows from Lemma 3.1(1) and (3.15),By equations (4.19), (4.20) and (4.21) at the case that i = 0 , we obtain the followingequivalent statements:(1) Hom G n ( π p | G n , π q ) = 0 ;(2) Hom G n ( u ρ ( m , d − × (( e σ × π ′ p ) | G t ) , π q ) = 0 for any e σ ∈ S un ρ m not appearingin cupp Z ( π p ) ∪ cupp Z ( ν − / π q ) . (cid:3) Lemma 4.4.
We keep using notations in the previous lemma. We still assume that m + d ≥ m i + d i and m + d ≥ l j + e j for all i, j . Then Hom G n ( π p | G n , π q ) = 0 if and only if there exists k such that π q,k ∼ = u ρ ( m , d − , and for any e σ ∈ S un ρ m with e σ / ∈ cupp Z ( π p ) ∪ cupp Z ( ν − / π q ) , Hom G n ′ (( e σ × π ′ p ) | G n ′ , π ′ q ) = 0 , where n ′ = n − n ρ m ( d − and π ′ q = π q, × . . . π q,k − × π q,k +1 × . . . × π q,s .Proof. We first consider the if direction. Let e σ ∈ S un ρ m not appear in cupp Z ( π p ) ∪ cupp Z ( ν − / π q ) . By the hypothesis of if direction, e σ × π ′ p has a quotient π ′ q (where π ′ q isdefined as in the lemma). Hence, by exactness of parabolic induction, u ρ ( m , d − × ( e σ × π ′ p ) has a quotient π q,k × π ′ q ∼ = u ρ ( m , d − × π ′ q ∼ = π q . Thus, by the if part of Lemma 4.3, we obtain
Hom G n ( π p | G n , π q ) = 0 . We now consider the only if direction. Suppose
Hom G n ( π p | G n , π q ) = 0 . By using theonly if part of Lemma 4.3, we have that: Hom G n ( u ρ ( m , d − × (( e σ × π ′ p ) | G t ) , π q ) = 0 for some e σ ∈ S un ρ m not in cupp Z ( π p ) ∪ cupp Z ( ν − / π q ) . Here t = n − n ρ m ( d − .Recall that we are assuming m + d ≥ l + e ≥ l i + e i for any i . For any τ ∈ cupp( π q,i ) ,(1) if ( m + ( d − − ≥ ( l i + e i − , then τ is either not in cupp Z ( ρ ) or τ ∈ n ν − ( d − m − / ρ , . . . , ν ( d − m − / ρ o ; or(2) if ( m + ( d − − < ( l i + e i −
2) = ( m + d − , then τ is not in cupp Z ( ρ ) .Thus we can apply Proposition 4.2 to obtain that π ∼ = u ρ ( m , d − × ω for some irreducible Arthur type quotient ω of ( e σ × π ′ p ) | G t . Now by uniqueness of factor-ization of Arthur type representations in terms of Speh representations, there exists some k ∗ such that π q,k ∗ ∼ = u ρ ( m , d − , π q, × . . . × π q,k ∗ − × π q,k ∗ +1 × . . . × π q,s ∼ = ω. This proves the only if direction. (cid:3)
Theorem 4.5.
Conjecture 1.2 holds for non-Archimedean field F .Proof. We shall prove the reformulated problem in Section 1.2. Let π p and π q be Arthurtype representations of G n +1 and G n respectively. We can write as the product of Spehrepresentations i.e. π p = π p, × . . . × π p,r and π q = π q, × . . . × π q,s such that each π p,i (resp. π q,j ) is an (irreducible unitarizable) Speh representation u ρ i ( m i , d i ) (resp. u σ j ( l j , e j ) ). Let N ( π p , π q ) be the total number of factors π p,i and π p,j which are notcuspidal representation. The basic case is that all π p,i and π q,j are cuspidal representationsi.e. N ( π p , π q ) = 0 , and so π p and π q are generic. In that case, it is well-known from[JPSS83, GGP12].By [Ta86, Theorem 7.1], we may and shall assume that for ≤ i ≤ r , ≤ j ≤ s , m + d ≥ m i + d i and l + e ≥ l j + e j . We may also assume that m + d > or l + e > , and so either π p, or π q, is notcuspidal. Otherwise, it is the basic case.We now consider two cases: Case 1: m + d ≥ l + e , which implies m + d − + > l i + e i − for all i , and so ν / ν ( d + m − / ρ is in cupp( ν / π p, ) , but is not in the cuspidal support of any π q,i .Let π ′ p = π p, × . . . × π p,r . Let u = π p, ∼ = u ρ ( m , d ) . We first prove the only if direction and assume that Hom G n ( π p , π q ) = 0 . Using Lemma 4.4, there exists σ ∈ S un ρ m with σ / ∈ cupp Z ( ν − / π q ) and k ∗ such that π q,k ∗ = u − , and Hom G t ( σ × π ′ p , π ′ q ) = 0 , ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 17 where π ′ q = π q, × . . . × π q,k ∗ − × π q,k ∗ +1 × . . . × π q,s and t = n − n ρ m ( d − . Since σ × π ′ p is also an Arthur type representation with N ( σ × π ′ p , π ′ q ) = N ( π ′ p , π ′ q ) < N ( π p , π q ) , we can apply inductive hypothesis to obtain that σ × π ′ p ∼ = τ p, × . . . × τ p,k × τ − q, × . . . × τ − q,l π ′ q ∼ = τ − p, × . . . × τ − p,k × τ q, × . . . × τ q,l for some Speh representations τ p, , . . . , τ p,k , τ q, , . . . , τ q,l . Since the product is uniquelydetermined by the factors of those Speh representations [Ta86] and σ / ∈ cupp Z ( ν − / π ′ q ) ,we must have τ p,i ∗ ∼ = σ for some i ∗ . Since the products between Speh representationscommute, we may simply set i ∗ = 1 . With τ − p, = 1 , now we have π p ∼ = u × π ′ p ∼ = u × τ p, × . . . × τ p,k × τ − q, × . . . × τ − q,l ,π q ∼ = u − × π ′ q ∼ = u − × τ − p, × . . . × τ − p,k × τ q, × . . . × τ q,l as desired.Now we prove the if direction and so we consider π p ∼ = τ p, × . . . × τ p,k × τ − q, × . . . × τ − q,l and π q ∼ = τ − p, × . . . × τ − p,k × τ q, × . . . × τ q,l for some Speh representations τ p, , . . . , τ p,k , τ q, , . . . , τ q,l . From our choice of π p, and theassumption for Case 1, we must have that, by reindexing if necessary, τ p, ∼ = π p, . Then τ − p, ∼ = u ρ ( m , d − . This implies that τ p, × . . . × τ p,k × τ − q, × . . . × τ − q,l ∼ = π p, × . . . × π p,r = π ′ p , by unique factorization of Speh representations [Ta86]. Since N ( π ′ p , π ′′ q ) < N ( π p , π ′′ q ) ≤ N ( π p , π q ) , induction gives that for any σ of S un ρ m , Hom G n ′ ( σ × π ′ p , π ′′ q ) = 0 , where π ′′ q ∼ = σ − × τ − p, × . . . × τ − p,k × τ q, × . . . × τ q,l . Lemma 4.4 implies that Hom G n ( π p | G n , π q ) =0 as desired. Case 2: l + e > m + d , which implies l + e − + > m + d − . We use Proposition 4.1to obtain a unitarizable irreducible cuspidal representation σ of G so that Hom G n +1 ( π q × σ | G n +1 , π p ) = 0 ⇐⇒ Hom G n +1 ( π q × σ, π p ) = 0 ⇐⇒ Hom G n +1 ( σ ∨ × π ∨ q | G n +1 , π ∨ p ) = 0 ⇐⇒ Hom G n ( π p | G n , π q ) = 0 . Here π q , σ, π p are complex conjugate representations of π p , σ, π q respectively, and so thefirst ’if and only if’ implication is immediate. The second ’if and only if’ implication fromthat π p , π q , σ are unitarizable and so Hermitian self-dual, and the third one follows fromProposition 4.1.We also have that π q × σ is still an Arthur type representation. Note that N ( π q × σ, π p ) = N ( π p , π q ) . We now use the argument in Case 1 and inductive hypothesis to prove this case,where the role of π q, replaces the one of π p, . (cid:3) General cases: Bessel, Fourier-Jacobi and Rankin-Selberg models
In this section, we shall generalize the non-tempered GGP to other models of generallinear groups. We study some connections between models, which will be continued inSection 6.5.1.
Equal rank Fourier-Jacobi models.
Let S ( F n ) be the space of Bruhat-Schwartzfunctions on F n . For a character µ of G n , let ω µ, (resp. b ω µ, ) be a G n -representation withunderlying space S ( F n ) and the G n -action given by ( g.f )( v ) = µ ( g ) f ( g − v ) , (resp . ( g.f )( v ) = µ ( g ) f ( g t v )) . Let π ∈ Alg( G n ) . Since G n \ M n +1 ∼ = F n as topological spaces, and ω µν − / , ⊗ π canbe viewed as the space of smooth compactly-supported functions f : F n → µν − / π with G n acting by ( g.f )( v ) = g.f ( g − v ) , we have: µ ⊗ Λ( π ) | G n ∼ = ω µν − / , ⊗ π via the natural map for f ∈ Λ( π ) , f (cid:18) v f ( (cid:18) I n v (cid:19) ) (cid:19) . Set ζ F = ω ν − / , and set b ζ F = b ω ν / , . Proposition 5.1.
Let π, π ′ ∈ Alg( G n ) . Then there exists a character χ of F × such that χ / ∈ cupp Z ( π ) ∪ cupp Z ( ν − / π ′ ) and, for all i , Ext iG n (( χ × π ) | G n , π ′ ) ∼ = Ext iG n ( π ⊗ ζ F , π ′ ) . The assertion also holds if we replace for ζ F by b ζ F .Proof. By Lemma 3.2, → χ | M ¯ × π → ( χ × π ) | M → χ ¯ × ( π | M ) → . (5.22)Then χ | M ¯ × π ∼ = Λ( π ) by the definition of mirabolic induction. By using the above identi-fication, we have χ | M ¯ × π ∼ = π ⊗ ζ F . (5.23) ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 19
On the other hand, via Frobenius reciprocity, the condition that χ / ∈ cupp Z ( ν − / π ′ ) guar-antees that for all i , Ext iG n (( χ ¯ × ( π | M )) | G n , π ′ ) ∼ = Ext iG n (( ν / χ ) × ( π | G n − ) , π ′ ) = 0 . (5.24)Now standard long exact sequence argument on (5.22) with (5.23) and (5.24) gives, for all i , Ext iG n ( χ × π, π ′ ) ∼ = Ext iG n (( χ | M ¯ × π ) | G n , π ′ ) ∼ = Ext iG n ( π ⊗ ζ F , π ′ ) . The proof for b ζ F is similar. (cid:3) Bessel, Rankin-Selberg and mixed models.
Recall that ¯ ψ is a choice of a non-degenerate character on F . Let H = u x yh zu : u ∈ U m , u ∈ U m , h ∈ e G r +1 , x ∈ M at m × ( r +1) ,z ∈ M at ( r +1) × m , y ∈ M at m × m ⊂ G m + m + r +1 , and e G r +1 = { diag(1 , g ) : g ∈ G r } . We shall also write H B or H Bm ,m ,r for H .Let ϕ n : U n → C be a generic character on U n . Let ζ : H → C such that ζ ( u x yg zu ) = ϕ n ( u ) ϕ n ( u ) ¯ ψ ( x m , ) ¯ ψ ( z , ) ν ( g ) − ( m − m ) / , where x m , (resp. z , ) is the ( m , - (resp. (1 , -) coordinate of x (resp. z ). We shallalso sometimes write ζ B for ζ . Note that ν − ( m − m ) is the modular character of H .Let U ′ be the unipotent radical of H . The orbit by the conjugation action of ( T m +1 × G r × T m ) U ′ on φ is the unique dense orbit on the character space of U ′ , where T m +1 (resp. T m ) be the subgroup of diagonal matrices of G m +1 (resp. G m ), and as subgroupof H via embedding to the upper (resp. lower) corner. Remark 5.2. [GGP12, Sections 12 and 13] considers the space F r × F r equipped with theHermitian form: h ( x , y ) , ( x , y ) i = ( y t x , y t x ) ∈ F × F . Then the isometric subgroupof G n × G n on h , i is isomorphic to G n via projecting to the first factor. In analog toorthogonal group case, the Bessel subgroup defined [GGP12] is conjugate to H Bm,m,r , where r = n − m , for some m .When m = 0 or m = 0 , the model is sometimes called a Rankin-Selberg model[ChSu15, GS20]. We shall also write H Rm,r = H B ,m,r and ζ R = ζ B . (The matrix H Rm,r isconjugate to the one in Section 1.3.) When r = 0 , the model is Whittaker [Sh74], and when m = m = 0 , it is related to the restriction from G n +1 to G n in [AGRS10].There is another formulation of Bessel models, using Bernstein-Zelevinsky functors. Lemma 5.3.
Let π be a G r -representation, which extends to a H -representation trivially.Let n = m + m + r + 1 . Then there exist natural isomorphisms: u ind G n H Bm ,m ,r π ⊗ ζ B ⊗ ν ( m − m ) ∼ =(Φ + ) m +1 (Π m +1 ¯ × π ) | G n ∼ =(Φ + ) ( m + m +1) (Λ( π )) | G n ∼ = u ind G n H Rm m ,r π ⊗ ζ R ⊗ ν m + m Proof.
The second isomorphism follows from Proposition 3.5. Note that the last isomor-phism is a special case of the first isomorphism. It remains to prove the first isomorphism.Let w = diag( (cid:18) I r I m +1 (cid:19) , I m +1 ) . Using induction in stages, the subgroup from which (Φ + ) m +1 (Π m +1 ¯ × π )) | G n is induced, takes the form: Q ′ = g ∗∗ m ∗ ∗ ∗ u , where g ∈ G r , m ∈ G m and u ∈ U m , and so w − Q ′ w = H Bm ,m ,r .The conjugation by the element w then defines a map Γ from u ind G n H π ⊗ ζ B ⊗ ν m − m to (Φ + ) m +1 (Π m +1 ¯ × π )) | G n , as vector spaces, given by f (cid:18) g f ( w (cid:18) g (cid:19) ) (cid:19) Restricted to the unipotent subgroup U ′ of H B , Γ( f ) is copies of character ζ B , whilea function in (Φ + ) m +1 (Π m +1 ¯ × π ) restricted to U ′ is copies of another character in thesame B ′ -orbit as ζ B , where B ′ contains matrices of the form diag( I r , T l U l ) , where l = m + m . Hence there exists b ∈ B ′ such that the map f (cid:18) g f ( bw (cid:18) g (cid:19) ) (cid:19) is a G n -isomorphism.We also remark that the character ν / arising when restricted to G n cancels with thecharacter ν − / arising from the mirabolic induction in Π m +1 ¯ × π . (cid:3) The following result is proved by a similar method as in [GGP12], also see [ChSu15].
Proposition 5.4.
Let π , π be representations of G n and G r respectively. Let m + m + r + 1 = n . For any irreducible cuspidal representation σ of G m + m +2 such that and σ / ∈ cupp Z ( ν − / π ∨ ) ∪ cupp Z ( π ) , for all i , Ext iH Bm ,m ,r ( π ⊗ ζ B , π ∨ ) ∼ = Ext iG n ( σ × π , π ∨ ) . Remark 5.5.
Proposition 4.1 is a particular case of Proposition 5.4 for m = 0 , m = 0 and r = n . ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 21
Proof.
By Lemma 3.2 again, → σ | M ¯ × π → ( σ × π ) | M → σ ¯ × ( π | M ) → Since σ is cuspidal, σ | M ∼ = Π m + m +2 . Now with Proposition 3.5 and Lemma 5.3, u ind GH π ⊗ ζ B = ( σ | M ¯ × π ) | G n Similar argument with the proof of Proposition 5.1, one reduces to, for all i , Ext iG n ( σ × π , π ∨ ) ∼ =Ext iG n (( σ | M ¯ × π ) | G n , π ∨ ) ∼ =Ext iG n ( u ind G n H π ⊗ ζ B ⊗ ν m − m , π ∨ ) ∼ =Ext iG n ( π , u Ind G n H ( π ⊗ ζ B ) ∨ ) (taking duals) ∼ =Ext iH ( π , ( π ⊗ ζ B ) ∨ ) (Frobenius reciprocity) ∼ =Ext iH ( π ⊗ ζ B , π ∨ ) (taking duals)For the last three isomorphism, also see [Pr18]. (cid:3) We state the multiplicity one for the general cases (c.f. [GGP12]):
Corollary 5.6.
Let π be an irreducible representation in Alg( G n +1 ) and let π be anirreducible representation in Alg( G r ) . For any m , m , r with m + m + r = n , dim Hom H Bm ,m ,r ( π ⊗ ζ B , π ) ≤ , and for all i , dim Ext iH Bm ,m ,r ( π ⊗ ζ B , π ) < ∞ Proof.
Proposition 5.4 reduces to the case that restricting from G n +1 to G n , which is provedin [AGRS10] for Hom and follows from [Pr18, AS18] for higher Ext. (cid:3) Fourier-Jacobi models.
Let S ( F r ) be the space of Bruhat-Schwartz functions on F r . Let W = F r and let K r be the Heisenberg group i.e. K r is the group isomorphic to F ⊕ W ⊕ W ∨ with the multiplication: ( a, v, w ) · ( a ′ , v ′ , w ′ ) = ( a + a ′ + w t v ′ , v + v ′ , w + w ′ ) . Define H ′ r = w t ag v : v, w ∈ F r , a ∈ F, g ∈ G r and so H ′ r ∼ = G r ⋉ K r . Here we identify W and W ∨ with F r so that y ( x ) = y t x for x ∈ W and y ∈ W ∨ .Fix a character µ of G r . Let λ be a non-trivial character on F . The Weil representation ω µ,λ of K r associated to λ is the representation with underlying space as S ( W ) with theaction of K r given by: for f ∈ S ( W ) ∼ = S ( F r ) , (( a, v, w ) .f )( x ) = λ ( a − w t x − w t v ) f ( x + v ) . and for f ∈ S ( W ∨ ) ∼ = S ( F r ) , (( a, v, w ) .f )( y ) = λ ( a + y t v ) f ( y + w ) . This extends ω µ,λ to a H ′ r -representation e ω µ,λ (resp. b ω µ,λ ) given by: for g ∈ G r , and f ∈ S ( W ) (resp. f ∈ S ( W ∨ ) ), ( g.f )( x ) = µ ( g ) · f ( g − .x ) , (resp. ( g.f )( y ) = µ ( g ) · f ( g t .y ) ) . Lemma 5.7.
Let π ∈ Alg( G r ) , extend trivaily to H ′ r . Then π ⊗ b ω µ, ¯ ψ ∼ = u ind H ′ r H B , ,r µπ ⊗ ( ζ B ⊗ ν / ) . Proof.
We can identify ν − / π ⊗ b ω µ, ¯ ψ with the space of smooth compactly supported func-tions f : F r → ν − / π with the action given by ( g.f )( y ) = g.f ( g t y ) . Since H B , ,r \ H ′ r ∼ = F r as topological spaces, the identification gives a map F : π ⊗ b ω µ, ¯ ψ → u ind H ′ r H B , ,r µπ ⊗ ( ζ B ν / ) given by F ( f )( y ) = f ( y t I r ) (cid:3) Now we consider general Fourier-Jacobi models. Let m , m ≥ . Let H = H m ,m ,r and U H be the subgroup of G m + m + r containing all elements of the form: u x yh zu and u x yI r +2 zu with u ∈ U m − , u ∈ U m − , h ∈ H ′ r , x ∈ M at m − ,r +2 , z ∈ M at r +2 ,m − and y ∈ M at m − ,m − . We shall write H Fm ,m ,r or H F . Note that we have H ∼ = H ′ r ⋉ U H . In thecase that m = m = 1 , it recovers the notion for H ′ r .We now extend the representations ω µ,λ of H ′ r to be a representation of H , still denoted ω µ,λ by abuse of notation, whose underlying space is S ( F r ) with the action, for f ∈ S ( F r ) , u x yh zu .f = ϕ m ( u ) ϕ m ( u )( h.f ) . (5.25)We similarly define the representation b ω µ,λ Set ζ = ζ Fm ,m ,r,λ = ζ F = ν ( m − m ) / e ω ν − / ,λ , and b ζ = ζ Fm ,m ,r,λ = b ζ F = ν ( m − m ) / b ω ν / ,λ . Again when m = m , it is the original notion of Fourier-Jacobi model in [GGP12, Section15]. The restriction problems involving ζ F (and b ζ F ) (i.e. Hom H ( π ⊗ ζ F , π ) ) do notdepend on a choice of λ . Proposition 5.8.
Let n = m + m + r with m , m , r ≥ . u ind G n H Bm − ,m ,r π ⊗ ζ B ⊗ ν m − m +1 ∼ = u ind G n H Fm ,m ,r π ⊗ b ζ F ⊗ ν m − m . ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 23
Proof.
From constructions, ζ B | U H ∼ = b ζ F | U H . Note that H ′ r normalizes U H and the conju-gation action of H ′ r on b ζ F | U H is trivial. One can extend the identification in Lemma 5.7to, as H F -representations, π ⊗ ( b ζ F ⊗ ν − / ν ( m − m ) / ) ∼ = u ind H F H B π ⊗ ( ζ B ⊗ ν ( m − m +1) / ) . Now applying induction from H F to G , an induction by stages gives the lemma. (cid:3) Proposition 5.9.
Let m , m , r ≥ . Let n = m + m + r . Let π ∈ Alg( G n ) , and let π ∈ Alg( G r ) . Then, for any cuspidal representation σ of G n +1 − r with σ / ∈ cupp Z ( π ) ∪ cupp Z ( ν − / π ∨ ) , and for any i , Ext iH Fm ,m ,r ( π ⊗ b ζ F , π ∨ ) ∼ = Ext iG n ( σ × π , π ∨ ) Proof.
For all i , Ext iG n ( σ × π , π ∨ ) ∼ =Ext iG n ((Π n +1 − r ¯ × π ) | G n , π ∨ ) ∼ =Ext iG n ((Φ + ) n − r (Λ( π )) | G n , π ∨ ) (by Proposition 3.5) ∼ =Ext iG n ( u ind G n H Bm − ,m ,r π ⊗ ζ B ⊗ ν m − m +1 , π ∨ ) (by Lemma 5.3) ∼ =Ext iG n ( u ind G n H Fm ,m ,r π ⊗ b ζ F ⊗ ν m − m , π ∨ ) (by Proposition 5.8) ∼ =Ext iH Fm ,m ,r ( π ⊗ b ζ F , π ∨ ) The first isomorphism follows from a standard argument as before. The last isomorphismis similar to the proof of Proposition 5.4. (cid:3)
Now we give a connection of the two notions ζ F and b ζ F . Proposition 5.10.
Let m , m , r ≥ and let n = m + m + r . Let π ∈ Alg( G n ) and let π ∈ Alg( G r ) . For all i , Ext iH ( π ⊗ ζ F , π ∨ ) ∼ = Ext i e H ( θ ( π ) ⊗ b ζ F , θ ( π ) ∨ ) , where e H = H Fm ,m ,r = wθ ( H ) w − . Here w is the matrix with all in the antidiagonal and elsewhere.Proof. Let θ w be the action of θ followed by the conjugation of w . We use the same θ w for the induced map on representations. Note that θ w ( π ) ∼ = θ ( π ) as G n -representations, θ w ( π ∨ ) ∼ = θ ( π ∨ ) ∼ = θ ( π ) ∨ as G r -representation, and θ w ( ζ Fλ ) ∼ = b ζ Fλ − . (cid:3) Restrictions.Theorem 5.11.
Let ( H, ζ ) be any pair described in Sections 5.1, 5.2 and 5.3. Let π M and π N be Arthur type representations of G n and G r respectively. Then Hom H ( π M ⊗ ζ, π N ) = 0 if and only if their associated Arthur parameters M A and N A are relevant. Proof.
When r = 0 , the model is Whittaker and it is well-known. Assume r ≥ . For theBessel models, this follows from Proposition 5.4 (in which we choose σ to be a unitarizablecuspidal representation) and Theorem 4.5. For the Fourier-Jacobi models, using Propo-sitions 5.9 and 5.10, it is equivalent to show that θ ( π ) and θ ( π ) have relevant Arthurparameters. By the Gelfand-Kazhdan isomorphism [BZ76], θ ( π ) ∼ = π ∨ and θ ( π ) ∼ = π ∨ .Thus now the statement follows from that the duals of π , π have relevant Arthur param-eter if and only if π ∨ , π ∨ have relevant Arthur parameter. (cid:3) Fourier-Jacobi models and Bernstein-Zelevinsky theory
In Section 5, we apply Bernstein-Zelevinsky theory to obtain isomorphisms of models.In this section, we further investigate the isomorphisms, and a goal is to obtain Corollary6.4. We also discuss connections with previous results in [CS18b, Ch19].6.1.
Fourier-Jacobi model and its dual.
Recall that ζ F and b ζ F are defined in Sections5.1 and 5.3. We first consider the equal rank case. Proposition 6.1.
In the equal rank case, ζ F ∼ = b ζ F as G n -representations.Proof. Let a ∈ F × . For f ∈ S ( F r ) , define the Fourier transform: b f ( y ) = Z F r ¯ ψ ( ay t x ) f ( x ) dx, (6.26)which is still smooth and compactly supported, and so in S ( F r ) , and we regard it as a mapfrom ζ F to b ζ F . The well-definedness of the map follows from that, for f ∈ ζ F , c g.f ( y ) = Z F r ¯ ψ ( ay t x )( g.f )( x ) dx = ν − / ( g ) Z F r ¯ ψ ( ay t x ) f ( g − x ) dx = ν / ( g ) Z F r ¯ ψ ( ay t ( gx )) f ( x ) dx = ν / ( g ) Z F r ¯ ψ ( a ( g t y ) t x ) f ( x ) dx =( g. b f )( y ) . One can define the inverse similarly. (cid:3)
Remark 6.2.
We explain how Proposition 6.1 is compatible with left and right Bernstein-Zelevinsky filtrations in [CS18b] and [Ch19].Let χ be a ramified character of G . We consider the representation χ × n ∼ = 1 n × χ .Then we have two exact sequences: → χ | M ¯ × n → ( χ × n ) | M → χ ¯ × (1 n | M ) → as M -representation )0 → θ ( χ | M ¯ × n ) → ( χ × n ) | M t → χ ¯ × (1 n | M t ) → as M t -representation ) , where the last mirabolic induction is defined analogously as the one for M -representations.Note that, by definition, ζ F ∼ = ( χ | M × n ) | G n , b ζ F ∼ = θ ( χ | M × n ) | G n . Let λ = χ | M ¯ × n and b λ = θ ( χ | M ¯ × n ) . Now we will see that λ ∼ = b λ from results of [CS18b]and [Ch19]. ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 25
By using the above short exact sequence, we obtain that if s is the Bernstein componentnot containing the representation ( ν / χ ) × n − , then λ s ∼ = b λ s . Then it admits a Bernstein-Zelevinsky filtration with successive quotients, which weregard as G n -representations Ψ + ( χ × ν − / n − ) , Ψ + (1 n ) , Φ + Ψ + ( ν − / n − ) The only layers contributing to the Bernstein component containing ( ν ± / χ ) × n − is thefirst and third quotients.However, it follows from left and right Bernstein-Zelevinsky filtrations (see a similar caseof Steinberg representation in [CS18b]), or more directly from indecomposability of eachrestricted Bernstein components [Ch19], for any Bernstein-component containing χ × n − , ( χ × n ) s ∼ = C × n − , where C is the Bernstein component of S ( F ) containing χ . Now, by multiplicity onetheorem, χ × n | G n has unique quotients ν / × n − and ν − / × n − . Thus, we muststill have that ( χ | M ¯ × n ) t ∼ = θ ( χ | M ¯ × n ) t ∼ = C × n − as desired. (Here the isomorphisms follow from the short exact sequence: → C → C → χ ′ → , where χ ′ = ν ± / χ .) Proposition 6.3.
We use the Fourier-Jacobi models in Section 5.3 and the Fourier trans-form defined in (6.26). The map
Ω : S ( F r ) → S ( F r ) by f ( y b f ( − a − y )) defines a H ′ r -map from ζ F , ,r, ¯ ψ to b ζ F , ,r, ¯ ψ .Proof. Let h = cI r v and h ′ = wI r . With computation in Proposition 6.1,it remains to check: Ω( h.f )( y ) = Z F r ¯ ψ ( − y t x )( h.f )( x ) dx = Z F r ¯ ψ ( c ) ψ ( − y t x ) f ( x + v ) dx = ¯ ψ ( c ) ¯ ψ ( y t v ) Z F r ¯ ψ ( − y t x ) f ( x ) dx = ¯ ψ ( c ) ¯ ψ ( y t v ) b f ( − a − y )= ¯ ψ ( c ) ¯ ψ ( y t v )Ω( f )( y ) = ( h. Ω( f ))( y )Ω( h ′ .f )( y ) = Z F r ¯ ψ ( − y t x )( h ′ .f )( x ) dx = Z F r ¯ ψ ( − y t x ) ¯ ψ ( − w t x ) f ( x ) dx = Z F r ¯ ψ ( − ( y + w ) t x ) f ( x ) dx = b f ( − a − ( y + w ))= Ω( f )( y + w ) = ( h ′ . Ω( f ))( y ) One can define inverse similarly. (cid:3)
We summarize the identifications as follow:
Corollary 6.4.
Let π ∈ Alg( G r ) . For m , m , r ≥ , u ind G n H Fm ,m ,r π ⊗ ζ F ⊗ ν m − m ∼ = u ind G n H Fm ,m ,r π ⊗ b ζ F ⊗ ν m − m ∼ = u ind G n H Bm − ,m ,r π ⊗ ζ B ⊗ ν m − m +1 ∼ = u ind G n H Rm − m ,r π ⊗ ζ R ⊗ ν m + m − Proof.
Proposition 6.3 implies that, as H Fm ,m ,r -representations, π ⊗ ζ F ∼ = π ⊗ b ζ F andhence we obtain the isomorphism. Now the remaining isomorphisms follow from Lemma5.3. (cid:3) Remark 6.5.
As we have seen, there is a more direct connection via (5.8) and the firstisomorphism of Lemma 5.3: u ind G n H Fm ,m ,r π ⊗ b ζ F ⊗ ν m − m ∼ = (Φ + ) m +1 (Π m ¯ × π )) | G n , (6.27)and similarly, we can obtain: u ind G n H Fm ,m ,r π ⊗ ζ F ⊗ ν m − m ∼ = (Φ + ) m (Π m +1 ¯ × π )) | G n . (6.28)The LHS of (6.27) and (6.28) are connected via Fourier transform in Proposition 6.3, whilethe RHS of (6.27) and (6.28) can be directly connected via Bernstein-Zelevinsky theory(Proposition 3.5). Corollary 6.6.
Let m , m , r ≥ . Let π ∈ Alg( G m + m + r ) and let π ∈ Alg( G r ) . Thereare natural isomorphisms: Ext iH Fm ,m ,r ( π ⊗ ζ F , π ∨ ) ∼ = Ext iH Fm ,m ,r ( π ⊗ b ζ F , π ∨ ) ∼ = Ext iH Bm − ,m ,r ( π ⊗ ζ B , π ∨ ) ∼ = Ext iH Rm m − ,r ( π ⊗ ζ R , π ∨ ) Examples on Fourier-Jacobi models.
In this subsection, we consider the equalrank Fourier-Jacobi model in Section 5.1.
Remark 6.7.
We identify S ( F r ) with S ( F ) ⊗ . . . ⊗ S ( F ) . We extend the Fourier-Jacobimodel π ⊗ ζ F to M n +1 -representation by letting V n (see Section 2.1) acting by translationon ζ F . For a compactly-supported set C in F , we define ch C to be the characteristicsfunction on C . Then the image of any element of the form ch C − (vol( C ) / vol( C ))ch C is zero in Ψ − ( π ⊗ ζ F ) . Since vol( gC ) = ν ( g )vol( C ) , Ψ − ( π ⊗ ζ F ) ∼ = π. Similarly, we also have that Φ − ( π ⊗ ζ F ) = π | M . ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 27
Example 6.8.
Let ∆ = [ ν − ( n − / , ν ( n − / ] for n ≥ . Let π = h ∆ i = 1 n . We con-sider the equal rank Fourier-Jacobi model in Section 5.1. Using Remark 6.7, as M n +1 -representations, it has a Bernstein-Zelevinsky filtration with successive quotients: Ψ + ( π ) , Φ + Ψ + ( h [ ν − ( n − / , ν ( n − / − ] i ) , Hence, ζ F , as G n -representation, has an irreducible quotient isomorphic to ν / (which isnot an Arthur type representation). On the other hand, ζ F admits a quotient by those func-tions f which vanishes at ∈ F r . Such quotient is isomorphic to ν − / , and is contributedfrom the layer Φ + Ψ + ( h [ ν − ( n − / , ν ( n − / − ] i ) . One can also deduce such quotient from identifying ζ F ∼ = b ζ F , and then extending b ζ F to M tn +1 -representation, using another form of Bernstein-Zelevinsky filtration [CS18b], [Ch19]as shown in the proof of Proposition 6.1. Example 6.9.
We again consider the equal rank Fourier-Jacobi model. For a generalizedSteinberg representation
St(∆) of G n , we expect that St(∆) ⊗ ζ F is projective and isisomorphic to the Gelfand-Graev representation of G n (c.f. [CS18b, CS19, Ch19]).7. Ext-branching laws
Conjecture on Ext-branching laws.
We formulate the following question aboutExt-branching laws stated in the form of a conjecture, which gives a possible generalizationof some observations in [GGP19].
Conjecture 7.1.
Let π M and π N be Arthur type representations of G n +1 and G n respec-tively. Then, for any i , Ext iG n ( π M , π N ) ∼ = M k Ext iG n +1 − k ( π [ k ] M , ( k − π N ) . It would be an interesting question to give a more precise formulation on Arthur param-eters to predict non-vanishing Ext-groups (see [GGP19, Proposition 5.7, Remark 5.8]).We remark that the appearance of left derivatives in the second spot comes from thesecond adjointness property of an induction in the Bernstein-Zelevinsky filtration (see e.g.[CS18b, Lemma 2.4]). We shall give few examples of the above conjecture below.7.2.
Hom -branching.Example 7.2.
Let π M and π N be generic Arthur type representations of G n +1 and G n respectively. Then π M = St( m ) and π N = St( n ) for some multisegments m and n . Acomputation via comparing cuspidal support gives that, for i = 0 or k = n + 1 , Ext iG n ( π [ k +1] M , ( k ) π N ) = 0 . Then
Hom G n ( π M , π N ) ∼ = Hom G ( π [ n +1] M , ( n ) π N ) ∼ = C . This recovers the Ext-vanishing theorem [Pr18, CS18b] and the multiplicity one theorem[AGRS10, SZ12] in this special case.
We remark that the same formulation of Conjecture 7.1 for arbitrary respective genericrepresentations π M and π N of G n +1 and G n is not true. Example 7.3.
Let π M and π N be Arthur type representations of G n +1 and G n respectively.Suppose their associated Arthur parameters are relevant. Write those Arthur parameters M A and N A as (1.2) and (1.3) respectively.Then Hom G n +1 − k ( π [ k +1] M , ( k ) π N ) = 0 ⇐⇒ k = r X d =0 dim M + d − s X d =0 dim M − d . (7.29)One should be able to deduce that from [Gu18] in which derivatives of Arthur type repre-sentations are studied, while we do not find a direct result. We sketch how to modify theproof of Theorem 4.5 to see (7.29). We use all the notations in the proof of Theorem 4.5,and in particular, write π M = π p = π p, × . . . × π p,r , and π N = π q = π q, × . . . × π q,s . The basic case is again all π p,i , π q,j are cuspidal, which is included in Example 7.2. Sincetaking duals behave well with derivatives, Case 2 (in Theorem 4.5) follow from Case 1.We only consider Case 1. Again, we use the short exact sequence: → π p, | M ¯ × π ′ p → π p | M → π p, ¯ × ( π ′ p | M ) → . Note that any Bernstein-Zelevinsky layer of π p, × ( π ′ p | M ) cannot contribute a non-zero Homwith π , by comparing cuspidal support. With similar consideration as in Theorem 4.5, theonly Bernstein-Zelevinsky layer that can contribute non-zero Hom with π takes the form ( ν − / π − p, ) ¯ × (Π × π ′ p ) , which can then be transferred to study the layers in ( ν − / π − p, ) ¯ × (( σ × π ′ p ) | M ) . Now one applies induction on the unique layer in ( σ × π ′ p ) | M that can contributenon-zero Hom with π q , which gives the required integer in (7.29).7.3. Generic representations.Theorem 7.4.
Let π M and π N be Arthur type representations of G n +1 and G n respectively.Suppose at least one of π M or π N is generic. Then there exists at most one integer j ∗ suchthat Ext iG n ( π [ j ∗ ] M , ( j ∗ − π N ) = 0 and furthermore if π M (resp. π N ) is not generic, then j ∗ (resp. j ∗ − ) is the level of π M (resp. π N ); and if both π and π are generic, then j ∗ = n + 1 .Proof. Assume that π is not a generic representation and π is a generic representation.Let π M = π p, × . . . × π p,r , π N = π q, × . . . × π q,s . where each π p,i is a Speh representation and each π q,j is isomorphic to St(∆ q,j ) for somesegment ∆ q,j .Then the i -th derivative π [ i ] M takes the form, for i + . . . + i r = i , ν / ( π ( i ) p, × . . . × π ( i r ) p,r ) ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 29
For each representation ω , we call the cuspidal support cupp( ω ) is(1) G -positive (resp. G -negative) if for each irreducible unitarizable cuspidal represen-tation σ and for all positive (resp. negative) integer a , the multiplicity of ν a σ in cupp( ω ) is at least that of ν − a σ .(2) balanced if cupp( ω ) is both G -positive and G -negative.Write π p,j = u ρ ( m, d ) . Note that for any i such that π ( i ) p,j is non-zero, cupp( ν / π ( i ) p,j ) =cupp( π − p,j ) + cupp(St(∆)) for ∆ = [ ν ( m − d ) / k ρ, ν ( m + d − / ρ ] , where k = i/n ρ . Since cupp( π − p,j ) is balanced and cupp(St(∆)) is G -positive, ν / π ( i ) p,j is G -positive for and i andis balanced only if i is the level of π p,j .On the other hand, since π q,j is a generalized Steinberg representation, ( i − π q,j is G -negative for all i and is balanced only if i = 0 or i is the level of π q,j . Thus cupp( π [ i ]1 ) =cupp( ( i − π ) only if i is the level of π as desired.Other cases are similar, or one may use Lemma 4.1. (cid:3) Remark 7.5.
Let π M and π N be as in Theorem 7.4. Assume π M is generic. Let ( π N ) − be the shifted highest derivative of π N and let π gen be the unique generic representationwith the same cuspidal support as ( π N ) − . In view of the above proof, one may expectthat Ext iG n ( π M , π N ) = 0 if and only if π M ∼ = π gen × π ′ for some generic representationof π ′ . While proving the if direction requires more discussion on computing Ext-groups,we will not carry out here. For some related computations, see, for example, temperedrepresentations [OS12] and Speh representations from Koszul resolution [Ch16].7.4. Another example.
One can obtain different information from various filtrations onrestricted representations [Pr93, CS18b, Ch19] such as left and right Bernstein-Zelevinskyfiltrations [CS18b, Ch19]. We shall see another example below using combinations of fil-trations:
Example 7.6.
Let ∆[ d ] = [ ν − ( d − / , ν ( d − / ] . For e ≥ , let π = h ∆[ e ] i × St(∆[ e − × σ, and let π = St(∆[ e − × h ∆[ e − i , where σ is a ramified character.We first investigate possible Bernstein-Zelevinsky layers contributing non-zero Ext-groups.Consider the derivatives: ( i ) h ∆[ e ] i × ( i ) St(∆[ e − × ( i ) σ and St(∆[ e − ( j ) × h ∆[ e − i ( j ) and, by comparing cuspidal supports, we must have i = 1 . Then we have the followingtwo possibilities: either(1) j = e − ; or(2) j = 1 ; orIn the case that j = e − , by comparing cuspidal support, we have j = 0 , and then i = e − . In the case j = 1 , we have two possibilities: (1) j = 0 , i = 0 .(2) j = e − , i = e − Now we find a cuspidal representation σ ′ as in Proposition 4.1 to consider the represen-tation π × σ ′ . Now we observe that there is two layers ( π × σ ′ ) | M that contribute non-zeroExt-groups (after restricting to G ): Now ( j , j ) = ( e − , , it contributes one layer λ := h ∆[ e − i × Π e +1 and ( j , j ) = (0 , , it contributes one layer λ := St(∆[ e − × h ν − / ∆[ e − i × Π and ( j , j ) = ( e − , , it contributes one (reducible) layer λ := λ = h ν − / ∆[ e − i × ν ( e − / × Π e +1 . We remark that λ is indecomposable as h ν / ∆[ e − i × ν − ( e − / is indecomposable.We now consider the dual restriction problem in Proposition 4.1, and so we consider therestriction for π × σ ′ for some cuspidal representation σ ′ of G .Using the following short exact sequence (Lemma 3.2): → h ∆[ e − i| M ¯ × (St(∆[ e − × σ ′ ) → ( π × σ ′ ) | M → h ∆[ e − i ¯ × ((St(∆[ e − × σ ′ ) | M ) → , and letting X ∗ = h ∆[ e − i| M ¯ × (St(∆[ e − × σ ′ ) ,X ∗ admits a filtration, in which there is one successive quotient isomorphic to λ andanother successive quotient isomorphic to λ .Using Bernstein-Zelevinsky filtration, we obtain a filtration on ( π × σ ′ ) | M of the form Y e ⊂ Y e − ⊂ . . . ⊂ Y = ( π × σ ′ ) | M . so that(1) Y e /Y e +1 ∼ = ( π × σ ′ ) ( e +1) ¯ × Π e +1 , and(2) Y e +1 is a simple module which is not isomorphic to any simple composition factorof λ , λ , λ , and(3) Y e /Y e +1 admits a filtration with one quotient isomorphic to λ and another quotientisomorphic to λ .The key of two filtrations is to obtain the following filtration, as M n +2 , and the directsum in the quotient roughly contributes the direct sum of Ext-groups in Conjecture 7.1: → I → X ∗ + Y e → X ∗ /I ⊕ Y e /I → , where I = X ∗ ∩ Y e . Let β := h n ν − / ∆[ e − , ν ( e − / o i × Π e +1 , which has multiplicity one in π × σ ′ | M . With the above information on X ∗ and Y e ,we can obtain further structure on I . The multiplicity forces that I contains the uniquecomposition factor β , but the indecomposability of λ also forces I contains the compositionfactor β , and a count on multiplicities gives that other composition factor of I is not ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 31 isomorphic to λ , λ or β (those are all the possible factors contributing non-zero Ext).Thus, we have that, for all k , Ext kG n +1 ( I | G n +1 , π ) = Ext kG n +1 ( λ | G n +1 , π ) = 0 . Then we have that
Ext kG n +1 ( π × σ ′ , π ) ∼ =Ext kG n +1 (( X ∗ + Y e ) | G n +1 , π ) ∼ =Ext kG n +1 ( X ∗ /I, π ) ⊕ Ext kG n +1 ( Y e /I, π ) ∼ =Ext kG n +1 ( λ , π ) ⊕ Ext kG n +1 ( λ , π ) ∼ =Ext kG n − (( π [1]2 , (2) π ) ⊕ Ext kG n +1 − e ( π [ e − , ( e ) π ) The first isomorphism follows from that the quotients by X ∗ + Y e has zero Ext by lookingat the possible composition factors and some computations on comparing cuspidal supports.The fourth isomorphism follows from the adjointness of the functors (see [CS18b, Lemma2.1] for more discussions).Since π ∨ ∼ = π and π ∨ ∼ = π , taking duals and using Proposition 4.1 gives that Ext kG n ( π , π ) ∼ = Ext kG n − ( π [2]1 , (1) π ) ⊕ Ext kG n +1 − e ( π [ e ]1 , ( e − π ) . The last isomorphism follows from [CS18b, Lemma 2.2].8.
Product preserving extensions
A motivating example in this and next section is the following. Let σ be an irreduciblecuspidal representation of G n . Let π and π be two admissible representations of G k suchthat the cuspidal supports of irreducible composition factors of π and π do not contain σ . Then, a simple application of Frobenius reciprocity and geometric lemma gives that Hom G n + k ( σ × π , σ × π ) ∼ = Hom G n ( σ, σ ) ⊠ Hom G k ( π , π ) ∼ = Hom G k ( π , π ) . Our goal is to generalize the above isomorphism to a larger class of examples in a functorialway, which is Theorem 9.1.8.1.
Preserving extensions.
Let S k be the set of all isomorphism classes of cuspidalrepresentations of G k , and let S = ⊔ k ≥ S k .Let C ⊂ S . Define
Alg C ( G m ) to be the full subcategory of Alg( G m ) whose objects π have finite lengths and satisfy the property that for any simple composition factor π ′ of π ,and for any σ ∈ cupp( π ′ ) , σ lies in C . For an irreducible cuspidal representation ρ of some G k , define cupp Z ( ρ ) = { ν n ρ } n ∈ Z . Theorem 8.1.
Fix an irreducible unitarizable cuspidal representation ρ of some G k . Let C = C m,d,ρ = n ν − ( d + m − / ρ, . . . , ν ( d + m − / ρ o ∪ ( S \ cupp Z ( ρ )) ⊂ S . Let π ∈ Alg C ( G n ) with length . Then π is indecomposable if and only if u ρ ( m, d ) × π isindecomposable. Note that ν − ( d + m − / ρ, . . . , ν ( d + m − / ρ are precisely all the (isomorphism classes of) cuspidal representations appearing in cuspidalsupport cupp( u ρ ( m, d )) .We will prove Theorem 8.1 in Section 8.6. One may also construct extensions betweentwo irreducible representations, and use this to give another proof of Theorem 8.1 for suchcase. In fact, that is also an original motivation for such formulation of Theorem 8.1.However, such approach will give a longer proof, but on the other hand, could possiblycover more cases (see Remark 8.3). Some hints on those constructions can be found fromthe study of [Ch18]. Remark 8.2.
In general, a product does not preserve extensions even if it preserves irre-ducibility. The standard example is that ν × (1 × ν ) , which is indecomposable of length .In this case, ν × h [1 , ν ] i and ν × St([1 , ν ]) are both irreducible, but ν × (1 × ν ) is semisimplewith two composition factors. Remark 8.3.
One may hope to improve the result from C m,d,ρ to a larger set of isomor-phism classes S \ n ν − ( d + m − / − ρ, ν ( d + m − / ρ o . Since we do not need such strong result for the application on branching laws, we will notprove that.8.2.
Faithfulness of a product.Lemma 8.4.
Let π ∈ Alg( G n ) . Let τ , τ ∈ Alg( G p ) . Then the functorial map (see Section9.1) Hom G p ( τ , τ ) → Hom G n + p ( π × τ , π × τ ) is injective.Proof. Let f ∈ Hom G p ( τ , τ ) . Then there is a surjection from τ to im f . Since parabolicinduction is exact, there is a surjection from π × τ to π × im f . Now the last surjection iszero ⇐⇒ π × im f = 0 ⇐⇒ im f = 0 . (cid:3) Jacquet functors.
Recall that N p is the subgroup of G n containing all matrices (cid:18) I n − p uI p (cid:19) , where u ∈ M at n − p,p .Let ∆ = [ ν a ρ, ν b ρ ] be a Zelevinsky segment. Let m = n ρ . Then [Ze80, Propositions 3.4and 9.5], the Jacquet functors are: h ∆ i N mi = [ ν a ρ, ν b − i ρ ] ⊠ [ ν b − i +1 ρ, ν b ρ ] . h ∆ i N − mi = h [ ν a + i ρ, ν b ρ ] i ⊠ h [ ν a ρ, ν a + i − ρ ] i St(∆) N mi = St([ ν a + i ρ, ν b ρ ]) ⊠ St([ ν a ρ, ν a + i − ρ ])St(∆) N − mi = St([ ν a ρ, ν b − i ρ ] ⊠ St([ ν b − i +1 ρ, ν b ρ ]) . ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 33
Note that computing π N − i is equivalent to first computing π N n − i to obtain a G i × G n − i -representation, then twisting by the action by the element (cid:18) I i I n − i (cid:19) to obtain a G i × G n − i -representation.8.4. Fully-faith product for large Speh.
Let e ∆( d, k ) = [ ν − ( d − / ρ, ν ( d − / k ρ ] . Wefirst consider e m ρ ( m, d, k ) = n ν − ( m − / e ∆( d, k ) , . . . , ν ( m − / e ∆( d, k ) o . Let e u ρ ( m, d, k ) = h e m ρ ( m, d, k ) i , which is sometimes called essentially Speh representationas it is a Speh representation twisted by a character. In particular, e u ρ ( m, d,
0) = u ρ ( m, d ) . Lemma 8.5.
Let π , π be admissible representations of G n . Fix ρ, d, m . For any k ≥ ,set e u k = e u ρ ( m, d, k ) . For k large enough, we have a natural isomorphism: Hom G n ( π , π ) ∼ = Hom G n + p ( e u k × π , e u k × π ) , where p = n ρ m ( d + k ) . Here naturality holds when the isomorphism holds for both π and π for the same k .Proof. We set k large enough such that ν ( d − m ) / k ρ is not in the cuspidal supports of anyirreducible representation of π and π .Let m = e m ρ ( m, d, k ) and let e u = e u ρ ( m, d, k ) . Using the injection: e u × π = h m i × π ֒ → ζ ( m ) × π , the left exactness of Hom G n + p ( e u × π , . ) gives Hom G n + p ( e u × π , ζ ( m ) × π ) ← ֓ Hom G n + p ( e u × π , e u × π ) (8.30)Let ∆ = [ ν ( − d + m ) / ρ, ν ( d + m − / k ρ ] . Since ζ ( m ) = h ∆ i × ζ ( m \ { ∆ } ) , Hom G n + p ( h m i × π , ζ ( m ) × π ) ∼ = Hom G n + p ( h m i × π , h ∆ i × π ′ ) , where π ′ = ζ ( m \ { ∆ } ) × π .Let q = n ρ m . Now Frobenius reciprocity gives that Hom G n + p ( h m i × π , h ∆ i × π ′ ) ∼ = Hom G q × G n + p − q (( h m i × π ) N n + p − q , h ∆ i ⊠ π ′ ) . Note that ν ( d + m − / k ρ does not appear in the cuspidal support of irreducible factors of π . With some analysis on Jacquet module from the geometric lemma (see, for example theproof of Lemma 8.8 below for more details), the only composition factor in ( h m i× π ) N n + p − q that has the same cuspidal support as h ∆ i ⊠ π ′ is h ∆ i ⊠ h m \ { ∆ }i × π . Thus we have
Hom( h m i × π , h ∆ i × π ′ ) ∼ = Hom( h m \ { ∆ }i × π , π ′ ) = Hom( h m ′ i × π , ζ ( m ′ ) × π ) , where m ′ = m \ { ∆ } , and so Hom( h m i × π , ζ ( m ) × π ) = Hom( h m ′ i × π , ζ ( m ′ ) × π ) Since ν ( d + m − / k − ρ does not appear in the cuspidal support of π ′ (when k ≥ ,otherwise we are done), we can repeat the similar process by replacing m \ { ∆ } with m .Inductively (which works by our choice of large k ), we obtain Hom G n + p ( h m i × π , ζ ( m ) × π ) ∼ = Hom G n ( π , π ) With (8.30),
Hom G n ( π , π ) ← ֓ Hom G n + p ( e u × π , e u × π ) . (8.31)Viewing e u × as a functor (also see Section 9.1), by Lemma 8.4, we have that Hom G n ( π , π ) ֒ → Hom G n + p ( e u × π , e u × π ) (8.32)Since we are dealing with admissible representations, the injections in (8.31) and (8.32)must be isomorphisms. Hence, we have that: Hom G n ( π , π ) ∼ = Hom G n + p ( e u × π , e u × π ) . (cid:3) Remark 8.6.
We remark that the above lemma does not require π and π to be in Alg C ( G n ) . In such case, e u ρ ( m, d, k ) × π may have more complicated structure. For example,when π has unique quotient, the cosocle of e u ρ ( m, d, k ) may not be irreducible. We give anexample here.Let ∆ = [ ν / , ν k ] for sufficiently large k . Let π = ν − / × ν / , which is reducible withlength . Then h ∆ i × π has the quotient h [ ν − / , ν k ] i × ν / since h ∆ i × ν − / has quotient h [ ν − / , ν k ] i , and hasthe quotient h ∆ i × St([ ν − / , ν / ]) , which is irreducible (deduced from similar way as in[Ch19, Appendix]), since π has the quotient St([ ν − / , ν / ]) .8.5. Product for irreducibility.
We use the notations in the previous section.
Lemma 8.7. [LM16]
Fix m, d and an irreducible unitarizable cuspidal representation ρ .Let m and m be multisegments with each segment ∆ satisfying ∆ ⊂ n ν − ( m + d − / ρ, . . . , ν ( m + d − / ρ o ∪ ( S \ cupp Z ( ρ )) . Then, for any k ≥ , (1) e u ρ ( m, d, k ) × h m i i ( i = 1 , ) is irreducible; (2) e u ρ ( m, d, k ) × h m i ∼ = e u ρ ( m, d, k ) × h m i if and only if m = m ; (3) e u ρ ( m, d, k ) × h m i i ∼ = h m i i × e u ρ ( m, d, k ) , for i = 1 , ; (4) Let p = n λ for λ = h m i i . Let N = N p and let N − = N − p . Then there is a unique (upto isomorphism) irreducible smooth G p -representation π such that e u ρ ( m, d, k ) ⊠ π (5) Suppose ω be an irreducible representation of G a + p . If e u ρ ( m, d, k ) ⊠ π is an irre-ducible quotient of ω N , then ω ∼ = e u ρ ( m, d, k ) × π . The statement also holds if wereplace ω N by ω N − and replace quotient by submodule. ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 35
Proof. (1) and (2) follow from [LM16, Corollary 6.7]. We only sketch how to deduce from[Ch19, Appendix]. Using a modified version of a lemma in [Ch19, Appendix], we have that ζ ( e m ρ ( m, d, k ) + m i ) ∨ ։ u ρ ( m, d, k ) × h m i i ֒ → ζ ( e m ρ ( m, d, k ) + m i ) , which forces that e u ρ ( m, d, k ) × h m i is the unique submodule of ζ ( e m ρ ( m, d, k ) + m i ) . (4) and(5) follow from Frobenius reciprocity and (2). (cid:3) Proof of Theorem 8.1.
We fix ρ, d, m . For simplicity, set e u k = u ρ ( m, d, k ) for k ≥ .Let ∆ k +1 = [ ν ( m − d ) / k +1 ρ, ν ( m + d − / k +1 ρ ] . Let C be as in Theorem 8.1 for such ρ , d and m . Lemma 8.8.
Let p = n ρ m . Let π ′ be an irreducible representation in Alg C ( G n ′ ) . Let n = n ′ + ( d + k + 1) mn ρ . There is a unique irreducible composition factor ω in (St(∆ k +1 ) × e u k × π ′ ) N − n − p which is isomorphic to St(∆ k +1 ) ⊠ τ for some irreducible τ of G n − p , and moreover, ω ∼ = St(∆ k +1 ) ⊠ ( e u k × π ′ ) . Proof.
For simplicity, set λ = e u k × π ′ , which is irreducible by Lemma 8.7. Note that ν ( m + d − / k +1 ρ is not in the cuspidal support of e u k × π ′ . To compute (St(∆ k +1 ) × λ ) N − n − p ,we first compute (St(∆ k +1 ) × λ ) N p (see discussions in Section 8.3), and then twisting the action by an element. Then geometriclemma on (St(∆ k +1 ) × λ ) N p yields a filtration successive quotients of the form St([ ν l +1 ρ, ν b ρ ]) × ω ⊠ St([ ν a ρ, ν l ]) × ω ′ . and this gives a filtration on ( e u k × St(∆ k +1 )) N − n − p with successive quotients taking the form St([ ν a ρ, ν l ρ ]) × ω ′ ⊠ St( ν l +1 ρ, ν b ρ ]) × ω. (8.33)Here ω and ω ′ are representations whose cuspidal supports do not contain ν ( m + d − / k +1 ρ .Thus an irreducible composition factor γ of (St(∆ k +1 ) × λ ) N − n − p can take the form St(∆ k +1 ) ⊠ τ only if l = b in (8.33). In such case, the successive quotient from geometric lemma isirreducible and is isomorphic to γ ∼ = St(∆ k +1 ) ⊠ λ . (cid:3) Lemma 8.9.
There exists a surjection from
St(∆ k +1 ) × e u k to e u k +1 .Proof. Let ∆ = ∆ k +1 . It follows from Lemma 2.1 that there is a surjection τ := St(∆) × St( ν − ∆) × . . . × St( ν − ( d + k ) ∆) → e u k +1 , and similarly, τ ′ := St( ν − ∆) × . . . × St( ν − ( d + k ) ∆) → e u k . This gives surjections τ = St(∆) × τ ′ ։ St(∆) × e u k ։ e u k +1 . By uniqueness of the irreducible quotient for τ , we then also have that St(∆) × e u k has thesame unique irreducible quotient as τ . (cid:3) Lemma 8.10.
Let K be the kernel of the surjection in Lemma 8.9. For any π in Alg C ( G n ′ ) and any π ′ in Alg C ( G n ′ ) , Hom( K × π, e u k +1 × π ′ ) = 0 . Proof.
Let ∆ = ∆ k +1 . We have the following short exact sequence: → K → St(∆) × e u k → e u k +1 → , which gives the short exact sequence: → K × π → St(∆) × e u k × π → e u k +1 × π → . Let N − = N − n ′ + n ρ m ( d + k ) . The Jacquet functor is exact and so we have another short exactsequence: → ( K × π ) N − → (St(∆) × e u k × π ) N − → ( e u k +1 × π ) N − → . (8.34)Now, by second adjointness of Frobenius reciprocity, we have a map St(∆) ⊠ ( e u k × π ) → ( e u k +1 × π ) N − . The map is indeed injective. This follows first from the case that π is irreducible by usingirreducibility of e u k × π (Lemma 8.7), and then lift to the general case by an inductiveargument using functoriality of Frobenius reciprocity. (One can also prove the map isinjective by directly computing the composition factors of ( e u k +1 × π ) N − taking the form St(∆) ⊠ τ , see the proof of Lemma 8.8.)Now by Lemma 8.8 and counting on composition factors, all irreducible compositionfactors of the form St(∆) ⊠ τ in (St(∆) × e u k × π ) N − are mapped onto ( e u k +1 × π ) N − underthe surjection map in (8.34).Thus there is no irreducible composition factor of ( K × π ) N − taking the form St(∆) ⊠ τ .On the other hand, for any irreducible π ′ , ( e u k +1 × π ′ ) N − has irreducible composition factorof the form St(∆) ⊠ τ , which can be deduced by an argument using Frobenius reciprocity.Hence, following from the exactness of Jacquet functor, we must have Hom( K × π, e u k +1 × π ′ ) = 0 . (cid:3) Proof of Theorem 8.1.
We keep using the above notations. Let π ∈ Alg C ( G n ) of length .We shall use backward induction to prove that, for any k ≥ , e u k × π is indecomposable, andmoreover e u k × π has unique irreducible quotient. When k is sufficiently large, Lemma 8.7implies that e u k × π has length 2, and Lemma 8.5 (and Lemma 8.7 (2)) imply the uniquenessof the quotient, which also then implies the indecomposability.Let π and π be the two irreducible composition factors of π . Let λ i = e u k × π i ( i = 1 , ). λ and λ are irreducible, and π ∼ = π ⇔ λ ∼ = λ by Lemma 8.7.Suppose e u k × π is not indecomposable. Let ∆ = ∆ k +1 . This gives an isomorphism e u k × π ∼ = λ ⊕ λ . and so there exists surjections, by Lemma 8.9, St(∆) × e u k × π ∼ = St(∆) × λ ⊕ St(∆) × λ → e u k +1 × π ⊕ e u k +1 × π ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 37
This implies that:(1) if λ = λ , then for both i = 1 , , Hom G (St(∆) × e u k × π, e u k +1 × π i ) = 0; (2) if λ ∼ = λ , then dim Hom G (St(∆) × e u k × π, e u k +1 × π ) ≥ . On the other hand, we have the following short exact sequence from Lemma 8.9: → K × π → St(∆) × e u k × π → e u k +1 × π → . By Lemma 8.10,
Hom( K × π, e u k +1 × π i ) = 0 for i = 1 , . Hence we have Hom( e u k +1 × π, e u k +1 × π i ) ∼ = Hom(St(∆) × e u k × π, e u k +1 × π i ) . However, by induction hypothesis and irreducibility of e u k +1 × π i , the former Hom hasdimension one for both i = 1 or if λ ∼ = λ , and has dimension one for precisely oneof i = 1 , if λ = λ . This gives a contradiction to (1) or (2) above. Thus e u k × π isindecomposable as desired, and since e u k × π has length , it also has unique irreduciblequotient. This completes the proof.9. Product functor of a Speh representation
Fully-faithful product.
Let C be as in Theorem 8.1. Let π ∈ Alg C ( G p ) . Define thefunctor × π, C = × π, C ,n : Alg C ( G n ) → Alg C ( G n + p ) as: × π, C ( ω ) = π × ω, and, for a map Ω : ω → ω in Alg C ( G n ) , × π, C (Ω)( f )( g ) = (Id π ⊠ Ω)( f ( g )) , where f ∈ u ρ ( m, d ) × ω is a smooth function f : G n + p → u ρ ( m, d ) ⊠ ω (Section 2.2). Theorem 9.1.
Let d, m be positive integers, and let ρ be an irreducible cuspidal represen-tation of some G k . Let C = n ν − ( d + m − / ρ, . . . , ν ( d + m − / ρ o ∪ ( S \ cupp Z ( ρ )) . Then the functor × u ρ ( m,d ) , C is fully-faithful.Proof. It suffices to check the conditions in Lemma 10.1. It follows from definition that
Alg C ( G k ) is Serre. Condition (1) is automatic. Condition (2) follows from Theorem 8.1.Conditions (3) and (4) follow from Lemma 8.7. (cid:3) It is possible to modify the proof of Theorem 8.1 to give another proof of Theorem 9.1without deducing from length case while length case is simpler. Let p = n ρ md . For π ∈ Alg C ( G n + p ) , define R u ρ ( m,d ) ( π ) = Hom G p ( u ρ ( m, d ) , π N − n ) , whichis regarded as a G n -representation by ( g.f )( u ) = g. ( f ( u )) , and is an object in Alg C ( G n ) .Now Hom
Alg C ( G n ) ( π ′ , R u ρ ( m,d ) ( π )) =Hom G n ( π ′ , R u ρ ( m,d ) ( π )) ∼ =Hom G p × G n ( u ρ ( m, d ) ⊠ π ′ , π N − ) ∼ =Hom G n + p ( u ρ ( m, d ) × π ′ , π )=Hom Alg C ( G n + p ) ( × u ρ ( m,d ) , C ( π ′ ) , π )) Thus × u ρ ( m,d ) is left adjoint to R u ρ ( m,d ) . Corollary 9.2.
Let u = u ρ ( m, d ) . Let π be in Alg C ( G n ) . Then π ∼ = R u ρ ( m,d ) ( u ρ ( m, d ) × π ) . Proof.
Since R u ρ ( m,d ) is right adjoint to × u ρ ( m,d ) , Theorem 9.1 implies that R u ρ ( m,d ) ◦× u ρ ( m,d ) is isomorphic to the identity functor (see e.g. [Sta, Lemma 4.24.3]). (cid:3) Corollary 9.2 also gives the following:
Corollary 9.3.
Let π ′ be in Alg C ( G n ) . Suppose π is an irreducible quotient of u ρ ( m, d ) × π ′ .Then π ∼ = u ρ ( m, d ) × ω for an irreducible quotient ω of π ′ . We need a stronger variation for Corollary 9.3:
Corollary 9.4.
Let C be as in Theorem 9.1. Let π be a (not necessarily admissible)representation of G n . Let π be in Alg C ( G n + p ) , where p = n ρ md . Then if π is a quotientof u ρ ( m, d ) × π , then there exists a non-zero quotient ω of π such that π ∼ = u ρ ( m, d ) × ω. In particular, if π is irreducible, then π ∼ = u ρ ( m, d ) × ω for an irreducible quotient ω of π . If π is an irreducible Arthur type representation, then π ∼ = u ρ ( m, d ) × ω for someirreducible Arthur type representation ω .Proof. Let u = u ρ ( m, d ) . By adjointness, we have =Hom G n + p ( u × π , π ) ∼ = Hom G n ( π , R u ( π )) , and let f be the map in Hom G n ( π , R u ( π )) corresponding to the surjection from u ρ ( m, d ) × π to π .Now using adjointness, we have the following commutative diagram: Hom G n + p ( u × ω, π ) ∼ = (cid:15) (cid:15) Hom G n + p ( u × π , π ) o o ∼ = (cid:15) (cid:15) Hom G n + p ( u × τ, π ) ∼ = (cid:15) (cid:15) o o o o Hom G n ( ω, R u ( π )) Hom G n ( π , R u ( π )) o o Hom G n ( τ, R u ( π )) o o o o , where the two horizontal rows are exact from the short exact sequence → ω = ker f → π → τ = im f → . ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 39
The leftmost Hom’s are zero. Thus when adjointness back, we get back the surjectivemap u × τ → π , and the injection im f ∼ = τ ֒ → R u ( π ) . Since π is in Alg C ( G n + p ) , τ is also in Alg C ( G n ) . Now by comparing the number ofcomposition factors on R u ( u × τ ) ∼ = τ (Corollary 9.2) and that on R u ( π ) , we must have u × τ ∼ = π .It remains to prove the last statement. Suppose π ∼ = u ρ ( n, d ) × ω is an Arthur typerepresentation. Then π and u ρ ( n, d ) being Hermitian self-dual implies that ¯ ω ∨ × u ρ ( n, d ) ∼ = ¯ π ∨ ∼ = π ∼ = u ρ ( n, d ) × ω ∼ = ω × u ρ ( n, d ) . This implies that ¯ ω ∨ ∼ = ω by Lemma 8.7 and so it is Hermitian self-dual. Thus ω isunitarizable by a result of Bernstein [Be84, Corollary 8.1]. Now the classification [Ta86]of unitarizable representations and unique factorization gives that ω is an Arthur typerepresentation. (cid:3) Appendix: Some homological algebra
Let A = Alg( G l ) . Let B = Alg( G n ) . Via Yoneda extension, any element in Ext A ( X, Y ) corresponds to a short exact sequence in A . Then, for an exact functor F , F sends ashort exact sequence to a short exact sequence, and this defines a map from Ext A ( X, Y ) to Ext B ( F ( X ) , F ( Y )) . Lemma 10.1.
Let C be a full Serre subcategory of A = Alg( G l ) . Let F : C → B be anexact additive functor. Let B = Alg( G n ) and let D be a Serre full subcategory of B . Wealso regard objects in C as objects in A via the inclusion. Assume that (1) any object in C is of finite length; (2) for any simple objects X, Y in C , the induced map of F , from Ext A ( X, Y ) to Ext B ( F ( X ) , F ( Y )) is an injection and, (3) F ( X ) is a simple object in D if X is simple in C ; and (4) for any simple objects X and Y in C , F ( X ) ∼ = F ( Y ) if and only if X ∼ = Y .Then for any objects X, Y in C , the induced map from Ext A ( X, Y ) to Ext B ( F ( X ) , F ( Y )) is also injective, and F : C → D is fully-faithful i.e.
Hom B ( F ( X ) , F ( Y )) ∼ = Hom D ( F ( X ) , F ( Y )) ∼ = Hom C ( X, Y ) ∼ = Hom A ( X, Y ) for any objects X, Y in C .Proof. Let X and Y be objects in C . When both lengths of X and Y are in C , Hom D ( F ( X ) , F ( Y )) ∼ = Hom C ( X, Y ) , Ext A ( X, Y ) ֒ → Ext B ( F ( X ) , F ( Y )) are guaranteed by (2), (3) and (4). We first fix the length of X to be at most some n . Weshall prove the statement for arbitrary Y by induction on the length of Y . For an object Y in C , let Y be an irreducible quotient of Y . Then we have a short exactsequence: → Y → Y → Y → . Since C is Serre, Y and Y are in C .Note that we have the following commutative diagram: Hom A ( X, Y ) / / (cid:15) (cid:15) Ext A ( X, Y ) / / (cid:15) (cid:15) Ext A ( X, Y ) / / (cid:15) (cid:15) Ext A ( X, Y ) (cid:15) (cid:15) Hom B ( F ( X ) , F ( Y )) / / Ext B ( F ( X ) , F ( Y )) / / Ext B ( F ( X ) , F ( Y )) / / Ext B ( F ( X ) , F ( Y )) , where the horizontal maps come from long exact sequences, in which the connecting homo-morphism is the Yoneda product, and vertical maps for Ext are described in the beginningof this section, and the vertical map for Hom is the map induced from the functor.We have the first vertical arrow is isomorphism and the second and forth vertical arrowsare injections by induction hypothesis. Then it is direct to check that the third verticalarrow is also an injection.Now we consider another commutative diagram: / / Hom A ( X, Y ) / / (cid:15) (cid:15) Hom A ( X, Y ) / / (cid:15) (cid:15) Hom A ( X, Y ) / / (cid:15) (cid:15) Ext A ( X, Y ) (cid:15) (cid:15) / / Hom B ( F ( X ) , F ( Y )) / / Hom B ( F ( X ) , F ( Y )) / / Hom B ( F ( X ) , F ( Y )) / / Ext B ( F ( X ) , F ( Y )) The first and third vertical arrows are isomorphisms by induction and the last verticalarrow is an injection by induction again. Thus we have that the second vertical arrow isan isomorphism.Now we switch the role of X and Y , and use similar argument to prove that the assertionis true for X and Y of arbitrary finite length. (cid:3) References [AGRS10] A. Aizenbud, D. Gourevitch, S. Rallis and G. Schiffmann,
Multiplicity one theorems , Ann. ofMath. (2) (2010), no. 2, 1407-1434.[AS18] A. Aizenbud and E. Sayag, Homological multiplicities in representation theory of p-adic groups.Math. Z. (2019), https://doi.org/10.1007/s00209-019-02262-4.[Ar89] J. Arthur, Unipotent automorphic representations: conjectures, Orbites unipotentes et represen-tations, II. Asterisque No. 171-172 (1989), 13-71.[At18] H. Atobe, The local theta correspondence and the local Gan-Gross-Prasad conjecture for thesymplectic-metaplectic case, Math. Ann. (1-2), 225-295 (2018)[Be84] J. Bernstein, P-invariant distributions on GL(N) and the classification of unitary representa-tions of GL(N) (non-Archimedean case), in Lie Group Representations, II (College Park, Md.,1982/1983),Lecture Notes in Math.1041, Springer-Verlag, New York, 1984, pp. 50-102[BZ76] I. N. Bernstein and A. V. Zelevinsky, Representations of the group
GL( n, F ) , where F is a non-archimedean local field, Russian Math. Surveys 31:3 (1976), 1-68.[BZ77] I. N. Bernstein and A. V. Zelevinsky, Induced representations of reductive p-adic groups , I, Ann.Sci. Ecole Norm. Sup. (1977), 441-472.[BP14] R. Beuzart-Plessis, Endoscopie et conjecture locale raffinee de Gan-Gross-Prasad pour les groupes’unitaires, Compos. Math. (2015), no. 7, 1309-1371. ON-TEMPERED GAN-GROSS-PRASAD CONJECTURE 41 [Ch16] K.Y. Chan, Duality for Ext-groups and extensions of discrete series for graded Hecke algebras,Adv. Math. (2016), 410-453.[Ch18] K.Y. Chan, Some methods of computing first extensions between modules of graded Hecke algebras,Algebra and Represent Theory , 859-895 (2018). http://doi.org/10.1007/s10468-017-9742-8[Ch19] K.Y. Chan, Homological branching law for (GL n +1 ( F ) , GL n ( F )) : projectivity and indecompos-ability, arXiv:1905.01668, submitted.[CS19] K.Y. Chan and G. Savin, Bernstein-Zelevinsky derivatives: a Hecke algebra approach, IMRN(2019), 731-760, https://doi.org/10.1093/imrn/rnx138[CS18] K.Y. Chan and G. Savin, the Iwahori-component of the Gelfand-Graev representations, Math. Z.(2018), 288, 125-133, https://doi.org/10.1007.s00209-017-1882-3.[CS18b] K.Y. Chan and G. Savin, A vanishing Ext-branching theorem for (GL n +1 ( F ) , GL n ( F )) ,arXiv:1803.09131, submitted.[ChSu15] F. Chen and B. Sun: Uniqueness of Rankin-Selberg Periods, International Mathematics ResearchNotices, 14, pp 5849-5873 (2015).[GP93] B. Gross and D. Prasad, On the decomposition of a representation of SO n when restricted to SO n − . Canad. J. Math. 44 (1992), no. 5, 974-1002.[GGP12] W.T. Gan, B.H. Gross, and D. Prasad, Symplectic local root numbers, central critical L values,and restriction problems in the representation theory of classical groups. Sur les conjectures deGross et Prasad. I., Asterisque No. 346 (2012), 1-109.[GGP19] W.T. Gan, B.H. Gross, and D. Prasad, Branching laws for classical groups: the non-temperedcase, arXiv:1911.02783v1.[GI16] W.T. Gan and A. Ichino, The Gross-Prasad conjecture and local theta correspondence, Invent.Math., (2016), no. 3, 705-799.[Gu18] M. Gurevich, On restriction of unitarizable representations of general linear groups and the non-generic local Gan-Gross-Prasad conjecture, arXiv: 1808.02640[GS20] D. Gourevitch and E. Sayag, Annihilator varieties of distinguished modules of reductive lie alge-bras, arXiv:2001.11746v2[HT01] M. Harris and R. Taylor, The geometry and cohomology of some simple Shimura varieties, Withan appendix by Vladimir G. Berkovich, Annals of Mathematics Studies, 151. Princeton UniversityPress, Princeton, NJ, 2001.[He00] G. Henniart, Une preuve simple des conjectures de Langlands pour GL(n) sur un corps p-adique.Invent. Math., 139(2):439-455, 2000.[Pr93] D. Prasad, On the decomposition of a representation of GL(3) restricted to
GL(2) over a p -adicfield, Duke Math. J. (1993), 167-177.[Pr18] D. Prasad, An Ext-analogue of branching laws, ICM proceedings 2018.[JPSS83] H. Jacquet, I. Piatetski-Shapiro and J. Shalika, Rankin-Selberg convolutions, Amer. J. Math. 105(1983), no. 2, 367-464.[LM14] Lapid, E. and A. Mínguez. On a determinantal formula of Tadić. Amer. J. Math. 136 (2014):111-142.[LM16] E. Lapid, A. Mínguez, On parabolic induction on inner forms of the general linear group over anon-Archimedean local field , Sel. Math. New Ser. (2016) 22, 2347-2400.[MW12] C. Mœglin and J.-L. Waldspurger, La conjecture locale de Gross-Prasad pour les groupes speciauxorthogonaux: le cas general, Sur les conjectures de Gross et Prasad. II, Astersque. No. (2012),167-216.[OS12] E.M. Opdam, M. Solleveld, Extensions of tempered representations, Geom. Funct. Anal. 23 (2013)664-714.[Sc13] P. Scholze, The Local Langlands correspondence for GL n over p-adic fields, Invent. math. (2013)192: 663. https://doi.org/10.1007/s00222-012-0420-5[Sh74] J. A. Shalika, The Multiplicity One Theorems for GL n , Annals of Mathematics Second Series,Vol. 100, No. 2 (Sep., 1974), pp. 171-193.[Sta] The Stacks Project Authors, Stacks Project, https://stacks.math.columbia.edu, 2020[SZ12] B. Sun and C. B. Zhu, Multiplicity one theorems: the Archimedean case , Ann. of Math. (2) (2012), no. 1, 23-44.[Ta86] M. Tadić, On the classification of irreducible unitary representations of GL(n) and the conjecturesof Bernstein and Zelevinsky, Ann. Sci. École Norm. Sup., (1986), 335-382.[Ve05] A. Venkatesh, The Burger-Sarnak method and operations on the unitary dual of GL(n). Repre-sentation Theory, vol. 9, pages 268-286 (2005).[Wa12] J.L. Waldspurger, La conjecture locale de Gross-Prasad pour les representations temperes desgroupes speciaux orthogonaux, Asterisque
No. (2012), 103-165. [Ze80] A. Zelevinsky,
Induced representations of reductive p-adic groups II , Ann. Sci. Ecole Norm. Sup. (1980), 154-210. Shanghai Center for Mathematical Sciences, Fudan University
E-mail address ::