Theta distinguished representations, inflation and the symmetric square L-function
aa r X i v : . [ m a t h . R T ] D ec THETA DISTINGUISHED REPRESENTATIONS, INFLATION AND THESYMMETRIC SQUARE L -FUNCTION EYAL KAPLAN
Abstract.
Let Π be a representation of a group H . We say that a representation τ is ( H, Π ) -distinguished, if it is a quotient of Π . It is natural to ask whether this notion“inflates” to larger groups, in the sense that a representation I ( τ ) induced from τ and H to a group G , is ( G, Π ) -distinguished. We study representations distinguished by thetarepresentations: H = GL n , Π is a pair of the exceptional representations of Kazhdan andPatterson, G = GSpin n + and Π is a pair of the small representations of Bump, Friedbergand Ginzburg. We prove a Rodier-type hereditary property: a tempered representation τ isdistinguished if and only if I ( τ ) is distinguished, and the multiplicity in each model is thesame. If τ is supercuspidal and distinguished, we prove that the Langlands quotient of I ( τ ) is distinguished. As a corollary, we characterize supercuspidal distinguished representations,in terms of the pole of the local symmetric square L -function at s = Introduction
Let τ be an admissible representation of GL n ( F ) , where F is a local non-Archimedeanfield. Let θ and θ ′ be a pair of exceptional representations of a double cover ̃ GL n ( F ) of GL n ( F ) , in the sense of Kazhdan and Patterson [KP84]. We say that τ is distinguished ifHom GL n ( F ) ( θ ⊗ θ ′ , τ ∨ ) ≠ . (1.1)Here τ ∨ is the representation contragradient to τ . Equivalently, the space of GL n ( F ) -invariant trilinear forms on τ × θ × θ ′ is nonzero.This space first appeared in a global context. Let π be a unitary cuspidal automorphicrepresentation of GL n ( A ) , where A is the ad`eles ring of a global field. Assume that π hasa trivial central character. Bump and Ginzburg [BG92] proved that if the partial symmet-ric square L -function L S ( s, π, Sym ) has a pole at s =
1, the following period integral isnonvanishing ∫ Z GL n ( F )/ GL n ( A ) ϕ π ( g ) Θ ( g ) Θ ′ ( g ) dg. (1.2)Here Z is a subgroup of finite index in the center C GL n ( A ) of GL n ( A ) ( Z = C GL n ( A ) when n is odd); ϕ π is a cusp form in the space of π ; Θ and Θ ′ are automorphic forms in the spaceof a global exceptional representation of ̃ GL n ( A ) . For n =
2, an earlier work by Pattersonand Piatetski-Shapiro [PPS89] showed that a similar integral characterizes the pole at s = L -functions and to questionsof functoriality. Ginzburg, Jiang and Soudry [GJS10] described such relations in a generalsetup and also considered several examples. Let G n = GSpin n + be the odd general spingroup of rank n +
1. Let E ( g ; ρ, s ) be the Eisenstein series corresponding to an element ρ in the space of the representation of G n ( A ) induced from τ ∣ det ∣ s ⊗ s ∈ C , g ∈ G n ( A ) ).The residual representation E π is the space spanned by the residues E / (⋅ ; ρ ) of E ( g ; ρ, s ) at s = /
2. The following result formulated in [GJS10] (Theorem 3.3) was proved in a seriesof works ([BG92, BFG03, GJS10, Kap]): the following conditions are equivalent.(1) L S ( s, π, Sym ) has a pole at s = E π is nonzero.(4) π is the Langlands functorial transfer of an irreducible generic cuspidal automorphicrepresentation of the split SO n ( A ) (if n is even) or Sp ( n − )/ ( A ) ( n is odd).Note that this was stated in [GJS10] with G n replaced by SO n + (see below).To prove that the nonvanishing of (1.2) implies the nontriviality of E π , one can relate theperiod to the following co-period integral CP( E / (⋅ ; ρ ) , Θ , Θ ′ ) = ∫ C Gn ( A ) G n ( F )/ G n ( A ) E / ( g ; ρ ) Θ ( g ) Θ ′ ( g ) dg. Here Θ and Θ ′ belong to the exceptional, or small, representation of Bump, Friedberg andGinzburg [BFG03], whose analog for G n was described in [Kap14a]. This integral was studiedin [Kap] for SO n + (extended in [Kap14a] to G n ) and we proved the implication (2) ⇒ (3). Inthe setting of G n , one can also study the twisted symmetric square L -function. The Rankin-Selberg integral representation for this function has recently been developed by Takeda[Tak14].The global unfolding of CP ( E / (⋅ ; ρ ) , Θ , Θ ′ ) (in [Kap]) has a local counterpart. Assumethat τ is a distinguished representation. Let I ( τ ) = Ind G n Q n ( δ / Q n τ ∣ det ∣ / ⊗ ) , where Q n is theSiegel parabolic subgroup. Using an integral over Q n ( F )/ G n ( F ) , we show (Proposition 4.1)that for a certain pair θ and θ ′ of exceptional representations of ̃ G n ( F ) ([BFG03, Kap14a],see below), Hom G n ( F ) ( θ ⊗ θ ′ , I ( τ ) ∨ ) ≠ . (1.3)In other words, I ( τ ) is a distinguished representation of G n ( F ) . In fact, depending on thecentral character of τ , we may need to replace τ ∣ det ∣ / ⊗ τ ∣ det ∣ / ⊗ η where η is acharacter of F ∗ . (See the proposition for details.) The local problem is to prove that theLanglands quotient LQ ( I ( τ )) of I ( τ ) is also distinguished.In the present study we consider an irreducible unitary supercuspidal τ . In this caseI ( τ ) is either irreducible and generic, or is of length two, has a unique irreducible genericsubrepresentation and LQ ( I ( τ )) is non-generic. Here is our main result, implying that if τ is distinguished, so is LQ ( I ( τ )) . For more details see Corollary 4.4. Theorem 1.
The space of θ ⊗ θ ′ as a representation of G n ( F ) does not afford a Whittakerfunctional. A similar “inflation” phenomena has already been observed by Ginzburg, Rallis and Soudry[GRS99a] (Theorem 2). Assume that τ is a supercuspidal and self-dual representation, suchthat the exterior square L -function L ( s, τ, ∧ ) has a pole at s =
0. This implies that τ hasa Shalika model and then according to Jacquet and Rallis [JR96], τ admits a (nontrivial)GL n × GL n invariant functional. In turn, the representation parabolically induced from τ ∣ det ∣ / to Sp n has an Sp n × Sp n invariant functional. Ginzburg, Rallis and Soudry [GRS99b](Theorems 16-17) showed that an irreducible generic representation of Sp n does not admitsuch a functional. It follows that the Sp n × Sp n functional factors through the Langlandsquotient. This inflation was one of the ingredients used by Lapid and Mao for the proof of theirconjecture on Whittaker-Fourier coefficients, in the case of the metaplectic group ([LM13,LM14a, LM14b, LM15]). Note that their conjecture actually applies to any quasi-split groupas well as the metaplectic group. Our results here and their extension, in a forthcoming work,to an arbitrary irreducible generic distinguished representation, are expected to be used ina proof of this conjecture for even orthogonal groups.Here we have a similar relation between distinguished representations and the symmetricsquare L -function. Theorem 2.
Let τ be an irreducible unitary supercuspidal representation of GL n . Then τ is distinguished if and only if L ( s, τ, Sym ) has a pole at s = . Refer to Shahidi [Sha92] (Theorem 6.2) for a description of the poles of L ( s, τ, Sym ) inthis setting.The proof of Theorem 1 is essentially the local analog of the global computation of theco-period in [Kap, Kap14a]. Write Q n = M n ⋉ U n and let C = C U n be the center of theunipotent radical U n . The global unfolding argument involves a Fourier expansion of Θ along C . Consider a non-generic character of C , this means that its stabilizer in M n contains aunipotent radical V of a parabolic subgroup of GL n . The corresponding Fourier coefficientis constant on V , then the cuspidality of π is used to prove that the integral vanishes.When n is odd (and n > C are non-generic. In the even case thereis one generic orbit of characters. Its stabilizer is “almost” a Jacobi group, its reductivepart is Sp n / × G , where Sp n / is a symplectic group in n variables. One might attempt toprove invariance of the product of Fourier coefficients under Sp n / ( A ) , then use the fact that π does not admit nontrivial symplectic periods ([JR92]). Albeit the Fourier coefficients donot enjoy this invariance, a certain convolution against Weil theta functions, introduced byIdeka [Ike94], can be used instead.The local argument involves the computation of the twisted Jacquet modules θ C,ψ k of θ with respect to C and a representative ψ k of some orbit of characters. In contrast withthe global setting, the generic character is the crux of the proof. Roughly, this is because θ C,ψ k is a tensor of an exceptional representation of ̃ GL k ( F ) and the Jacquet module of anexceptional representation of ̃ G k ( F ) along C U k and a generic character.When the character is generic, the twisted Jacquet module is (by restriction) a repre-sentation of a Jacobi group. The local theory of smooth representations of Jacobi groups[vD78, MVW87, BS98] describes such a representation as a tensor κ ⊗ ω ψ , where ω ψ is theWeil representation. The Heisenberg group acts trivially on the space of κ , while the actionof the reductive part separates into an action on the space of κ , and one on the space of ω ψ .We prove that κ is a trivial representation of Sp n / ( F ) . Theorem 3.
Assume n is even and let ψ n / be a generic character of C U n . As a Jacobirepresentation θ C Un ,ψ n / is the direct sum of (possibly infinitely many) copies of ω ψ . Note that the action of the G part of the stabilizer on θ C Un ,ψ n / is given simply by thecentral character of θ .This result underlies Theorem 1. Furthermore, it implies the following multiplicity prop-erty. Assume that τ is irreducible and tempered. We prove that τ is distinguished if and onlyif I ( τ ) is. Moreover, the dimensions of (1.3) and (1.1) are equal. See Proposition 4.6 for theprecise statement. Note that Kable [Kab01] conjectured, and under a certain homogeneity EYAL KAPLAN assumption proved ([Kab01] Corollary 6.1), that (1.1) enjoys multiplicity one. These resultsmotivate the introduction of “exceptional models” over GL n and G n .Theorem 3 may have additional applications. Explicit descriptions of Jacquet modules ofexceptional representations have had numerous applications (see below).Note that when n = Q n is the Heisenberg parabolic subgroup in the notation of Ganand Savin [GS05]. In their terminology, Theorem 3 shows that θ is “weakly minimal”. Inthis case it is the minimal representation and θ C Un ,ψ n / ≅ ω ψ ([GS05] Section 3).Bump, Friedberg and Ginzburg [BFG03] constructed the small representation θ SO n + forthe special odd orthogonal group. This is a representation of a “double cover” ̃ SO n + ( F ) ,obtained by restricting the 4-fold cover of SL n + ( F ) of Matsumoto [Mat69]. For the lowrank cases n = ,
3, it is the minimal representation. In fact for n =
3, this representationwas already developed by Roskies [Ros96], Sabourin [Sab96] and Torasso [Tor97]. For n > B n ([Vog81]).Bump, Friedberg and Ginzburg [BFG06] showed that when n > θ SO n + is attached to oneof the possible coadjoint orbits, which is smallest next to the minimal one. This translatesinto the vanishing of a large class of Fourier coefficients, called generic in [BFG03] (see also[CM93, Car93, Gin06]). Locally, this means that a large class of twisted Jacquet modulesvanish. The representation θ SO n + was used by these authors to construct a lift with certainfunctorial properties, between covers of orthogonal groups [BFG06]. The representation θ SO was used to construct an integral representation ([BFG00]).There is a technical issue when working with ̃ SO n + ( F ) : the underlying field must containall 4 4-th roots of unity. This can be remedied using G n . Indeed, one obtains a nontrivialdouble cover of G n ( F ) by restricting the 2-fold cover of Spin n + ( F ) of Matsumoto [Mat69].The theory of Bump, Friedberg and Ginzburg [BFG03] can be extended to ̃ G n ( F ) , mainlybecause both groups are of type B n and in particular, the unipotent subgroups are isomor-phic. The details were carried out in [Kap14a]. Our results here are stated for G n , but applysimilarly to SO n + and θ SO n + (see Section 5).Minimal representations have been studied and used extensively, by numerous authors.Knowledge of Fourier coefficients, or Jacquet modules, has proved very useful for applications[GRS03, Gin06, GJS11]. They have played a fundamental role in the theta correspondence,the descent method and Rankin-Selberg integrals. See, for example [Vog81, Kaz90, KS90,Pra93, Sav93, BK94, Sav94, GRS97a, BFG00, GRS01, KPW02, BFG03, JS03, GS05, Sou06,LS08, LS10, GRS11].We mention that Bump and Ginzburg [BG92] also developed a local theory, where theyconsidered a similar space of equivariant trilinear forms, except that θ ′ was replaced by acertain induced representation. For n =
3, Savin [Sav92] determined the dimension of (1.1)for an arbitrary irreducible quotient of a principal series representation. He also conjectured(and proved for n = n ( F ) is precisely those representations, which are lifts from a certain prescribed classical group.Kable [Kab02] proved that these lifts are distinguished, the other direction was proved in[Kap14b].The group G n has been the focus of study of a few recent works, among which are theworks of Asgari [Asg00, Asg02] on local L -functions, Asgari and Shahidi [AS06, AS11] onfunctoriality and Hundley and Sayag [HS12] on the descent construction.The rest of this work is organized as follows. In Section 2 we provide notation and defi-nitions. In particular we describe the construction of exceptional representations. Section 3 contains the proof of Theorem 3. Theorems 1 and 2 are proved in Section 4. Section 5provides the formulation of our results for SO n + . Acknowledgments.
I wish to express my gratitude to Erez Lapid for suggesting this projectto me. I would like to thank Eitan Sayag, for explaining to me how to use his results withOffen ([OS08], Proposition 1). Lastly, I thank Jim Cogdell for his kind encouragement anduseful remarks. 2.
Preliminaries
General notation.
Let F be a local non-Archimedean field of characteristic differentfrom 2. For an integer r ≥
1, let µ r be the subgroup of r -th roots of unity in F . Set F ∗ r = ( F ∗ ) r . We usually fix a nontrivial additive character ψ of F . Then the normalizedWeil factor ([Wei64] Section 14) is denoted by γ ψ ( γ ψ ( a ) is γ F ( a, ψ ) of [Rao93], γ ψ (⋅) = r is ( , ) r . If G is a group, C G denotes its center. For x, y ∈ G and Y < G , x y = xyx − and x Y = { x y ∶ y ∈ Y } . Hereby we omit references to the field, e.g.,GL n = GL n ( F ) .2.2. The group GL n and its cover. Fix the Borel subgroup B GL n = T GL n ⋉ N GL n ofupper triangular matrices, where T GL n is the diagonal torus. For any k , k ≥ k + k = n , denote by P k ,k the maximal parabolic subgroup whose Levi part is isomorphic toGL k × GL k . Its unipotent radical is Z k ,k = {( I k zI k )} . The “mirabolic” subgroup P ○ n − , isthe subgroup of GL n of matrices with the last row ( , . . . , , ) . Let I n be the identity matrixof GL n and J n be the matrix with 1 on the anti-diagonal and 0 elsewhere. For g ∈ GL n , t g is the transpose of g .We will use the metaplectic double cover ̃ GL n of GL n , as constructed by Kazhdan andPatterson [KP84]. Let ̃ SL n + be the double cover of SL n + of Matsumoto [Mat69] and let σ SL n + be the corresponding cocycle of Banks, Levi and Sepanski [BLS99] (Section 3). Wedefine a 2-cocycle σ GL n of GL n via σ GL n ( a, a ′ ) = σ SL n + ( diag ( det a − , a ) , diag ( det a ′− , a ′ )) . This cocycle is related to the cocycle σ n of GL n defined in [BLS99] by σ GL n ( a, a ′ ) = c ( det a, det a ′ ) σ n ( a ′ , a ) . In particular σ GL ( a, a ′ ) = ( a, a ′ ) .2.3. The group
GSpin n + . We start by defining the special odd orthogonal groupSO n + = { g ∈ SL n + ∶ t gJ n + g = J n + } . Select its Borel subgroup B SO n + = B GL n + ∩ SO n + .Let Spin n + be the simple split simply-connected algebraic group of type B n . It is thealgebraic double cover of SO n + . We will take the Borel subgroup, which is the preimage of B SO n + . The set of simple roots of Spin n + is ∆ Spin n + = { α i ∶ ≤ i ≤ n } , where α i = ǫ i − ǫ i + for 1 ≤ i ≤ n − α n = ǫ n .The group G n = GSpin n + is an F -split connected reductive algebraic group, which canbe defined using a based root datum as in [Asg02, AS06, HS12]. It is also embedded in G ′ n + = Spin n + as the Levi part of the parabolic subgroup corresponding to ∆ G ′ n + −{ α } (see [Mat09]). We adapt this identification, which is more natural for the purpose of covergroups, because we will obtain a cover of G n by restricting the cover of G ′ n + . EYAL KAPLAN
Let δ Q be the modulus character of a parabolic subgroup Q < G n .In the degenerate case G = GL .The Borel subgroup of G n is denoted B n = T n + ⋉ N n , where N n is the unipotent radical(the rank of the torus is n + ≤ k ≤ n , denote by Q k = M k ⋉ U k the standard maximalparabolic subgroup of G n with M k ≅ GL k × G n − k . This isomorphism is not canonical. Wedescribe the choice used in [Kap14a], which is convenient for certain computations (seebelow).The derived group SL k of GL k is generated by the root subgroups of { α i ∶ ≤ i ≤ k } .Let η ∨ , . . . , η ∨ k be the standard cocharacters of T GL k and map η ∨ i ↦ ǫ ∨ i + − ǫ ∨ for 1 ≤ i ≤ k .Regarding G n − k , the set { α i ∶ k + ≤ i ≤ n + } identifies G ′ n − k and if θ , . . . , θ n − k + are thecharacters of T n − k + , define θ ∨ ↦ ǫ ∨ and for 2 ≤ i ≤ n − k + θ ∨ i ↦ ǫ ∨ k + i .The projection G ′ n → SO n + is an isomorphism between unipotent subgroups, hence wecan identify unipotent subgroups of G n with those of SO n + .We use the character ǫ to define a “canonical” character Υ of G n . Namely Υ is theextension of − ǫ to G n (the only other choice would be to use ǫ ).The aforementioned embedding of GL k × G n − k in M k has a few properties, suitable forcomputations. To compute the conjugation of b ∈ GL k on U k , we can simply look at thisaction in SO n + , where b takes the form diag ( b, I ( n − k )+ , J kt b − J k ) . The image of G is C G n .The restriction of Υ to GL k is det.2.4. The double cover of
GSpin n + . Let ̃ G ′ n + be the double cover of G ′ n + , constructedby Matsumoto [Mat69] using ( , ) as the Steinberg symbol. Restricting the cover to G n , weobtain the exact sequence 1 → µ → ̃ G n p Ð→ G n → . Then ̃ G n is a double cover of G n . For a subset X ⊂ G n , ̃ X = p − ( X ) . Let s ∶ G ′ n + → ̃ G ′ n + bethe block-compatible section constructed by Banks, Levi and Sepanski [BLS99] and σ G ′ n + bethe corresponding cocycle. Denote the restriction of σ G ′ n + to G n × G n by σ G n . In [Kap14a]we proved that σ G n satisfies the following block compatibility property: if ( a, g ) , ( a ′ , g ′ ) ∈ GL k × G n − k ≅ M k , σ G n (( a, g ) , ( a ′ , g ′ )) = σ GL k ( a, a ′ ) σ G n − k ( g, g ′ )( Υ ( g ) , det a ′ ) . (2.1)We also mention that C ̃ G n = ̃ C G n .2.5. Representations.
Let G be an l -group ([BZ76] 1.1). Representations of G will becomplex and smooth. For a representation π of G , π ∨ is the representation contragradientto π . We say that π is glued from representations π , . . . , π k , if π has a filtration, whosequotients (which may be isomorphic or zero) are, after a permutation, π , . . . , π k .Regular induction is denoted Ind and ind is the compact induction. Induction is notnormalized.Let π be as above and let U < G be a unipotent subgroup, exhausted by its compactsubgroups (always the case here). Let ψ be a character of U . The Jacquet module of π with respect to U and ψ is denoted π U,ψ . It is a representation of the stabilizer of ψ (andnormalizer of U ). The action is not normalized. We have an exact sequence0 → π ( U, ψ ) → π → π U,ψ → . The kernel π ( U, ψ ) can be characterized by the Jacquet-Langlands lemma: Lemma 2.1. (see e.g. [BZ76] v in the space of π belongs to π ( U, ψ ) if andonly if ∫ O π ( u ) v ψ − ( u ) du = , for some compact subgroup O < U . When ψ =
1, we simply write π ( U ) and π U .Assume that ̃ G is a given r -th cover of G . Let ε ∶ µ r → C ∗ be a faithful character. Arepresentation π of ̃ G is called ε -genuine if it restricts to ε on µ r . When r =
2, such arepresentation is simply called genuine.Let ϕ ∶ G → ̃ G be a section and assume π and π ′ are representations of ̃ G , such that π is ε -genuine and π ′ is ε − -genuine. Then π ⊗ π ′ (outer tensor product) is a representation of G via g ↦ π ( ϕ ( g )) ⊗ π ′ ( ϕ ( g )) . The actual choice of ϕ does not matter, whence we omit it.2.6. Representations of Levi subgroups.
Levi subgroups of classical groups are directproducts. The tensor of representations of the direct factors is usually used, to describe theirrepresentations. In passing to a cover group, these factors do not necessarily commute andthen the tensor construction fails.Except for the case of k = n , the preimages of GL k and G n − k in ̃ M k do not commute. Thesame phenomena occurs in GL n . The following discussion describes a replacement for theusual tensor product. For more details see [Kap14a].The metaplectic tensor product in the context of GL n has been studied by various authors[FK86, Sun97, Kab01, Mez04, Tak13]. Our definitions were motivated by the constructionof Kable [Kab01] (see Remark 2.1 below).For any H < G n , put H ◻ = { h ∈ H ∶ Υ ( h ) ∈ F ∗ } . The subgroup H ◻ is normal in H andthe quotient is a finite abelian group. If ξ is a representation of ̃ H , let ξ ◻ = ξ ∣ ̃ H ◻ . Assume0 < k < n . According to (2.1), the preimages of GL ◻ k and G ◻ n − k are commuting in the cover.Then if ρ and π are genuine representations of ̃ H < ̃ GL k and ̃ H < ̃ G n − k , the representation ρ ◻ ⊗ π ◻ is a genuine representation of p − ( H ◻ × H ◻ ) ≅ {( ǫ, ǫ ) ∶ ǫ ∈ µ }/( ̃ H ◻ × ̃ H ◻ ) . Put H = H H and define I ◻ ( ρ, π ) = ind ̃ Hp − ( H ◻ × H ◻ ) ( ρ ◻ ⊗ π ◻ ) . When k = n , ̃ GL n and ̃ G are commuting, then the metaplectic tensor is defined as usual. Remark 2.1.
The arguments in [Kab01] do not readily extend to G n , mainly because C G n < G ◻ n for all n , and then C G n does not play a role similar to that of C GL n in the cover. We will use Mackey Theory to relate this induced representation to ρ and π . We reproducethe following result from [Kap14a], which mimics [Kab01] (Theorem 3.1) in our context. Lemma 2.2.
The representation I ◻ ( ρ, π ) is a direct sum of [ F ∗ ∶ F ∗ ] copies of ind ̃ Hp − ( H ◻ × H ) ( ρ ◻ ⊗ π ) . EYAL KAPLAN
Proof of Lemma 2.2.
Since p − ( H ◻ × H ◻ ) is normal of finite index in ̃ H and p − ( H ◻ × H ) modulo p − ( H ◻ × H ◻ ) is abelian, I ◻ ( ρ, π ) = ⊕ a ∈ H ◻ / H ind ̃ Hp − ( H ◻ × H ) ( ρ ◻ ⊗ ω a π ) . (2.2)Here ω a ( h ) = ( Υ ( h ) , det a ) ranges over the finite set of characters of H , which are trivialon H ◻ . By (2.1) if a ∈ ̃ GL k and h ∈ ̃ G n − k , a − h = ( Υ ( h ) , det a ) h . Hence ρ ◻ ⊗ ω a π = a ( a − ( ρ ◻ ) ⊗ π ) and the result follows. (cid:3) The Weil representation.
We introduce the Weil representation, which plays animportant role in this work. Let n = k and λ be the symplectic bilinear form on F n definedby λ ( u, v ) = u ( J k − J k ) t v , where u and v are regarded as rows. Let H n be the ( n + ) -dimensional Heisenberg group, with the group operation given by ( u , u ; z ) ⋅ ( v , v ; z ) = ( u + v , u + v , z + z + λ (( u , u ) , ( v , v ))) , where u i , v i ∈ F k and z i ∈ F .Let Sp k be the symplectic group defined with respect to λ , i.e., the group of g ∈ GL n such that λ ( ug, vg ) = λ ( u, v ) for all u, v ∈ F n . Let ̃ Sp k be the metaplectic double coverof Sp k , realized using the normalized Rao cocycle [Rao93]. The group Sp k acts on H n via g − ( u , u ; z ) g = (( u , u ) g ; z ) .Fix a nontrivial additive character ψ of F . Let ω ψ be the Weil representation of H n ⋊ ̃ Sp k ,realized on the space S ( F k ) of Schwartz-Bruhat functions on F k . Recall the followingformulas for ω ψ (see [Per81]): for ϕ ∈ S ( F k ) , ω ψ (( u , z )) ϕ ( x ) = ψ ( z ) ϕ ( x + u ) , (2.3) ω ψ (( , u ; 0 )) ϕ ( x ) = ψ ( xJ kt u ) ϕ ( x ) , (2.4) ω ψ ((( I k uI k ) , ǫ )) ϕ ( x ) = ǫψ ( xJ kt u t x ) ϕ ( x ) ( ǫ ∈ µ ) . (2.5)Let R = {( , u ; 0 )} < H n and U = {( I k uI k )} < Sp k . Since U normalizes (in fact, commuteswith) R , ( ω ψ ) R is a U -module. We will use the following simple observation. Claim 2.3.
The vector spaces ( ω ψ ) R and ( ω ψ ) RU are one dimensional.Proof of Claim 2.3. According to Lemma 2.1 and (2.4), the space of ω ψ ( R ) is S ( F k − ) .Hence ( ω ψ ) R is one dimensional. Then ( ω ψ ) RU = (( ω ψ ) R ) U is nonzero, because by Lemma 2.1and (2.5), a function ϕ ∈ S ( F k ) such that ϕ ( ) ≠ ω ψ ( RU ) . (cid:3) We will also encounter the tensor ω ψ ⊗ ω ψ − of two Weil representations. This is a repre-sentation of H n , trivial on C H n , and a representation of Sp k . Regarding it as a representationof C H n / H n , we can compute its twisted Jacquet modules. The group Sp k acts transitivelyon the nontrivial characters of C H n / H n , hence we can consider only one nontrivial character. Claim 2.4.
Let µ be a character of H n , which is trivial on C H n .(1) If µ = , ( ω ψ ⊗ ω ψ − ) H n ,µ is the trivial one-dimensional representation of Sp k .(2) If µ ( u , u ; z ) = ψ (( u ) ) , ( ω ψ ⊗ ω ψ − ) H n ,µ is the trivial one-dimensional representationof P ○ n − , ∩ Sp k . Proof of Claim 2.4.
First assume µ =
1. Equality (2.4) implies that elements ϕ ⊗ ϕ ′ inthe space of ω ψ ⊗ ω ψ − , such that the supports of ϕ and ϕ ′ (as functions in S ( F k ) ) aredifferent, vanish under the Jacquet module along R (use Lemma 2.1). Hence the spaceof ( ω ψ ⊗ ω ψ − ) R is isomorphic to S ( F k ) . Since the action of C H n is trivial on ω ψ ⊗ ω ψ − , ( ω ψ ⊗ ω ψ − ) R = ( ω ψ ⊗ ω ψ − ) RC Hn . Now RC H n is a normal subgroup of H n , whence ( ω ψ ⊗ ω ψ − ) H n is a quotient of ( ω ψ ⊗ ω ψ − ) RC Hn .It then follows from (2.5) that the action of U is trivial on ( ω ψ ⊗ ω ψ − ) RC Hn and in particular,on ( ω ψ ⊗ ω ψ − ) H n . The latter is a representation of Sp k , and because Sp k is generated (asan abstract group) by the subgroups w U , w ∈ Sp k (it is enough to take Weyl elements w ), ( ω ψ ⊗ ω ψ − ) H n must be a trivial representation of Sp k .Moreover ( ω ψ ⊗ ω ψ − ) H n is one-dimensional. This can be seen as follows. Replace afunction f ∈ S ( F k ) = ( ω ψ ⊗ ω ψ − ) RC Hn with its Fourier transform ̂ f ( x ) = ∫ F k f ( y ) ψ ( x ( t y )) dy .This changes the module structure, but the action of Sp k remains trivial. The action of ( u ,
0; 0 ) ∈ H n is now given by ( u ,
0; 0 ) ⋅ ̂ f ( x ) = ψ − ( x t u ) ̂ f ( x ) (instead of (2.3)). Next apply Lemma 2.1, the space of ( ω ψ ⊗ ω ψ − ) RC Hn (( RC H n )/ H n ) is S ( F k − ) .We conclude that ( ω ψ ⊗ ω ψ − ) H n is the trivial one-dimensional representation of Sp k .Now consider the case of the nontrivial µ , given in the statement of the claim. Since µ ∣ R = ( ω ψ ⊗ ω ψ − ) H n ,µ is a quotient of ( ω ψ ⊗ ω ψ − ) RC Hn and in particular, a trivial representationof U . In coordinates, P ○ n − , ∩ Sp k = ⎧⎪⎪⎪⎨⎪⎪⎪⎩⎛⎜⎝ u vg ∗ ⎞⎟⎠ ∶ g ∈ Sp k − ⎫⎪⎪⎪⎬⎪⎪⎪⎭ . (In particular it stabilizes µ .) We see that U < P ○ n − , ∩ Sp k and using conjugations of U byelements from Sp k − , it follows that ( ω ψ ⊗ ω ψ − ) H n ,µ is the trivial representation of P ○ n − , ∩ Sp k .To show this is a one-dimensional representation, argue as above using the Fourier trans-form, the space of ( ω ψ ⊗ ω ψ − ) RC Hn (( RC H n )/ H n , µ ) is S ( F k − (− , , . . . , )) . (cid:3) Exceptional or small representations.
We describe the exceptional representationsthat appear in this work. Kazhdan and Patterson [KP84] introduced and studied these repre-sentations for GL n . For G n these are essentially the small representations of Bump, Friedbergand Ginzburg [BFG03], who developed them using a cover of SO n + (see Section 5). Theirconstruction was extended to G n in [Kap14a].Let G be either GL n or G n . Let B be the Borel subgroup of G , T be the maximal torusand ∆ be the subset of simple roots. For α ∈ ∆, if α is a long root and n >
1, put l ( α ) = l ( α ) =
1. Denote by α ∨ the coroot of α .Let ξ be a genuine character of C ̃ T . We say that ξ is exceptional if ξ ( s ( α ∨ ( x l ( α ) ))) = ∣ x ∣ , ∀ α ∈ ∆ , x ∈ F ∗ . The character ξ corresponds to a genuine irreducible representation ρ ( ξ ) of ̃ T (when n > ̃ T is a 2-step nilpotent subgroup). Since ξ is exceptional, the representation Ind ̃ G ̃ B ( δ / B ρ ( ξ )) has a unique irreducible quotient θ , called an exceptional representation. Note that θ isadmissible. Occasionally, we use the notation θ G to record the group. We appeal to the following explicit description of C ̃ T : C ̃ T GL n = ̃ T n C ̃ GL n with T n = { t ∶ t ∈ T GL n } , C ̃ GL n = { z ⋅ I n ∶ z ∈ F ∗ e } , where e is 1 if n is odd, otherwise e = C ̃ T n + = C ̃ T GL n ̃ G ,and if T n + = T n G , C ̃ T n + = ̃ T n + C ̃ GL n (see [KP84] p. 57 and [Kap14a] Section 2.1.6). Notethat G ◻ = G . Furthermore, the cocycle σ G satisfies σ G ( z ⋅ I n , z ′ ⋅ I n ) = ( z, z ′ ) ⌈ n / ⌉ .The exceptional characters can be parameterized in the following way. Start by fixing acharacter ξ of p ( C ̃ T ) , trivial on p ( C ̃ G ) . This does not determine ξ uniquely. For GL n , take ξ = δ / B GL n . For G n take ξ whose restriction to p ( C ̃ T GL n ) is δ / B GL n ⋅ ∣ det ∣ ( n + )/ . Given that ξ is trivial on p ( C ̃ G n ) , this determines ξ uniquely for G n (see [Kap14a] Section 2.3.3 for anexplicit formula).Now for any given ξ , there is a character χ of F ∗ (called “determinantal character” in[BG92]) and a nontrivial additive character ψ of F such that ξ ( ǫ s ( td )) = ǫξ ( td ) χ ( Υ ( td )) γ ⌈ n / ⌉ ψ ( z ) , ∀ t ∈ T , d = z e ⋅ I n ∈ T, ǫ ∈ µ . (2.6)The corresponding exceptional representation will be denoted θ χ,ψ . Claim 2.5.
We have θ χ,ψ = χθ ,ψ , where on the right-hand side we pull back χ to a non-genuine character of ̃ G via g ↦ χ ( Υ ( p ( g ))) . Additionally, if ψ is another additive characterof F , θ χ,ψ = ηθ χ,ψ for some square trivial character η of F ∗ .Proof of Claim 2.5. The first assertion is clear. Write ψ ( x ) = ψ ( αx ) for some α ∈ F ∗ . Then γ ψ = η γ ψ , where η ( z ) = ( α, z ) . Since η is trivial on F ∗ and η ( Υ ( d )) = η ( det d ) = η ( z n ) = η ( z ) (the last equality is trivial if n is even, because then z ∈ F ∗ ), we obtain χ ( Υ ( td )) γ ⌈ n / ⌉ ψ ( z ) = ( η ⌈ n / ⌉ χ )( Υ ( td )) γ ⌈ n / ⌉ ψ ( z ) . Hence θ χ,ψ = θ η ⌈ n / ⌉ χ,ψ = η ⌈ n / ⌉ θ χ,ψ . (cid:3) Exceptional representations have a useful inductive property. Let θ be an exceptionalrepresentation of ̃ G n . Following the arguments of Bump, Friedberg and Ginzburg [BFG03](Theorem 2.3), we computed θ U k ([Kap14a]). For 0 < k < n , ( θ χ,ψ ) U k ⊂ I ◻ ( θ GL k ∣ ⋅ ∣ ( n − k − )/ χ,ψ , θ G n − k χ,ψ ) . (2.7)If k (resp. n − k ) is odd, the exceptional representation of ̃ GL k (resp. ̃ G n − k ) is unique only upto varying the character ψ , or multiplying χ by a square trivial character of F ∗ . In any case,the space on the right-hand side of (2.7) is unique, because by Claim 2.5, the exceptionalrepresentation obtained by such a change to ψ or χ , has the same restriction to ̃ GL ◻ k (resp. ̃ G ◻ n − k ) as the original representation.If k = n , ( θ χ,ψ ) U n = θ GL n ∣ ⋅ ∣ ( n − )/ χ,ψ ⊗ θ G χ, . (2.8)In [Kap14a] we did not find the precise exceptional representations appearing on the right-hand side of (2.8), but this is simple to obtain: since C ̃ T n + = C ̃ T GL n ̃ G , there is an exceptionalcharacter ξ of C ̃ T GL n such that w ξ = w ′ ξ ⊗ ξ ∣ ̃ G , where w and w ′ are the longest Weylelements in the Weyl groups of G n and GL n (see [Kap14a] Claim 2.18 for details). It remainsto write ξ using (2.6). Remark 2.2.
The reason for the imprecise result when k < n is the lack of a definition fora tensor product. These results are sufficient for our applications. For GL n , Kable proved a result similar to (2.8) for all standard unipotent radicals, withthe tensor replaced by his metaplectic tensor ([Kab01] Theorem 5.1 (4)).Exceptional representations enjoy the vanishing of a large class of twisted Jacquet modules.The following result is the extension of Theorem 2.6 of [BFG03] and Proposition 3 of [BFG06]to G n (this extension appeared in [Kap14a]). It was used in [BFG03, BFG06] (for SO n + )to deduce all vanishing properties.The unipotent radical U is abelian. A character ψ ( ) of U takes the form ψ ( ) (⎛⎜⎝ u ∗ I n − ∗ ⎞⎟⎠) = ψ ( ua ) , where a ∈ F n − is a column. The length of a is defined to be t aJ n − a . When ψ is fixed andclear from the context, we also refer to t aJ n − a as the length of ψ ( ) . Theorem 4.
If the length of ψ ( ) is nonzero, θ U ,ψ ( ) = . See [Kap14a] (Lemma 2.25) for the details.3.
Twisted Jacquet modules of θ G n In this section we describe the twisted Jacquet modules of θ = θ G n with respect to thecenter of U n . These modules appear in a filtration of θ as a ̃ Q n -module and will be used inSection 4.The group GL n , embedded in M n , acts on the set of characters of C = C U n with finitelymany orbits. Let ψ be a nontrivial additive character of F . For any 0 ≤ k ≤ ⌊ n / ⌋ , define acharacter of C by ψ k ( c ) = ψ ( k ∑ i = c n − i + ,n + i ) ( c is regarded as a ( n + ) × ( n + ) unipotent matrix in SO n + ). The stabilizer of ψ k in ̃ M n is ̃ St ψ k , with St ψ k = {( a z b ) ∶ a ∈ GL n − k , b ∈ Sp k } × G . Here Sp k is the symplectic group in 2 k variables, corresponding to a symplectic bilinear formdefined according to ψ k . Then θ C,ψ k is a representation of ̃ St ψ k ⋉ U n .Regarding GL n − k and Sp k as subgroups of St ψ k ,St ψ k = (( GL n − k × Sp k ) ⋉ Z n − k, k ) × G . ( Z n − k, k was given in Section 2.2.)We turn to the proof of Theorem 3. Namely, for n = k , θ C,ψ k is the direct sum of copiesof the Weil representation ω ψ . Here is an outline of the proof. The theory of smooth repre-sentations of Jacobi groups ([vD78, MVW87, BS98]) implies that any such representation,with a central character ψ , takes the form κ ⊗ ω ψ , where the Heisenberg group acts triviallyon the space of κ , and the action of the symplectic group separates into an action on thespace of κ , and one on the space of ω ψ . The vanishing properties of θ - Theorem 4, will showthat κ is trivial. Proof of Theorem 3.
Put k = n /
2. The image of G in G n is C G n . Hence St ψ k is the directproduct of a Jacobi group and G . Moreover, ̃ G = C ̃ G n (see Section 2.4) and because θ isan irreducible representation, ̃ G acts by the central character of θ . Therefore in the proofwe ignore the G part of St ψ k .We may replace ψ k by any character of C in the same GL n -orbit, since the Jacquetmodule will be isomorphic. For convenience, we redefine ψ k ( c ) = ψ ( ∑ ki = c i,n + + i ) . We usethe notation of Section 2.7. The stabilizer St ψ k is now the symplectic group defined withrespect to the form λ . The cover ̃ St ψ k is a nontrivial double cover. We have an epimorphism ℓ ∶ ̃ St ψ k ⋉ U n → ̃ Sp k ⋉ H n : ℓ (⎛⎜⎝ I n u z − t uJ n I n ⎞⎟⎠) = ( t uJ n ; 12 ( k ∑ i = z i,i − n ∑ i = k + z i,i )) ∈ H n ,ℓ ((⎛⎜⎝ g J nt g − J n ⎞⎟⎠ , ǫ )) = ( J nt g − J n , ǫ ) ∈ ̃ Sp k ( g preserves λ, ǫ ∈ µ ) . The kernel of ℓ is contained in the kernel of ψ k . Therefore we can regard θ C,ψ k as a genuinerepresentation of ̃ Sp k ⋉ H n . As such, it is isomorphic to κ ⊗ ω ψ (see e.g. [BS98] p. 28), where κ is a non-genuine representation on a space V , and the action is given by ( g, ǫ ) h ⋅ ( f ⊗ ϕ ) = κ ( g ) f ⊗ ω ψ (( g, ǫ ) h ) ϕ, g ∈ Sp k , h ∈ H n , f ∈ V , ϕ ∈ S ( F k ) . We must show that κ is trivial. Consider the subgroup Y = ⎧⎪⎪⎪⎨⎪⎪⎪⎩⎛⎜⎝ yI ( n − ) ⎞⎟⎠⎫⎪⎪⎪⎬⎪⎪⎪⎭ < Sp k . Since Sp k is generated by the conjugates x Y where x ∈ Sp k , it is enough to prove invarianceunder Y . That is, we show κ Y = κ. (3.1)A character of Y takes the form y ↦ ψ ( αy ) for some α ∈ F (on the left-hand side y isregarded as a matrix, on the right-hand side as an element of F ). The action of the torus ofSp k on the nontrivial characters of Y has finitely many orbits, namely the different squareclasses in F ∗ . Each of these orbits is open. Therefore the kernel of the Jacquet functor κ Y is filtered by representations induced from κ Y,ψ ( α ⋅ ) , where α ranges over the square classes([BZ76] 5.9-5.12). Hence (3.1) follows if we prove κ Y,ψ ( α ⋅ ) = α ≠ X = Y ⋅ {( , u ; 0 ) ∈ H n ∶ u = ( , . . . , , r )} (a direct product). Theepimorphism ℓ splits over X , hence there is a subgroup U < ̃ St ψ k U n isomorphic to X . In fact, U = ⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩⎛⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎝ y r − r / I n −
01 01 − r − yI n − ⎞⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎠⎫⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎬⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎭ < U . In [Kap] p. 25 it was incorrectly stated they are equal. The pullback of ψ ( α ⋅) to U is ψ ⋆ ( u ) = ψ (− αu ,n ) . Observe that ( θ C,ψ k ) U,ψ ⋆ =
0. Indeed, ( θ C,ψ k ) U,ψ ⋆ = θ CU,ψ k ψ ⋆ , which is a quotient of θ U ( C ∩ U ) ,ψ k ψ ⋆ . Since for u ∈ U ( C ∩ U ) , ψ k ψ ⋆ ( u ) = ψ (− αu ,n + u ,n + ) , any extension of ψ k ψ ⋆ to a character of U is a character of nonzero length.Thus Theorem 4 yields θ U ( C ∩ U ) ,ψ k ψ ⋆ = ( θ C,ψ k ) U,ψ ⋆ =
0. Therefore by Lemma 2.1,for any f ⊗ ϕ there is a compact O < Y ⋅ R ( R = {( , u ; 0 )} , see Section 2.7) such that ∫ O yr ⋅ ( f ⊗ ϕ ) ψ − ( αy ) dr dy = . (3.2)According to Claim 2.3 and Lemma 2.1 there is ϕ ∈ S ( F k ) such that for all compactsubgroups Y < Y and R < R , ϕ Y , R = ∫ Y ∫ R ω ψ ( yr ) ϕ dy dr ≠ . (3.3)Take f ∈ V . We will show that for large enough Y and R , ∫ Y κ ( y ) f ψ − ( αy ) dy ⊗ ϕ Y , R = . (3.4)This along with (3.3) imply that f belongs to the space of κ ( Y, ψ ( α ⋅)) .Plugging (3.3) into (3.4) and changing variables leads to ∫ Y ( ∫ Y ∫ R κ ( y ) f ⊗ ω ψ ( yry − ) ϕ ψ − ( αy ) dr dy ) dy . We will show that the inner drdy -integration vanishes for all y ∈ Y . Fix y . Since κ ∣ H n istrivial, this inner integration equals ∫ Y ∫ R yr ⋅ ( f ⊗ ω ψ ( y − ) ϕ ) ψ − ( αy ) dr dy. (3.5)Again resorting to Claim 2.3, ω ψ ( y − ) ϕ = c y ϕ + ϕ ○ y , (3.6)where c y ∈ C and ϕ ○ y belongs to the space of ω ψ ( R ) . Since y varies in a compact subgroup,there is a large enough R such that ∫ R ω ψ ( r ) ϕ ○ y dr = , ∀ y ∈ Y . (3.7)Furthermore (3.2) implies that for large Y and R , ∫ Y ∫ R yr ⋅ ( f ⊗ ϕ ) ψ − ( αy ) dr dy = c ∈ C , ∫ Y ∫ R yr ⋅ ( f ⊗ cϕ ) ψ − ( αy ) dr dy = . (3.8)Combining (3.6)-(3.8) we conclude that for sufficiently large R and Y , the inner drdy -integration (3.5) vanishes. Note the order of selecting the compact subgroups: first, choose R and Y which ensure (3.8), they depend only on f and ϕ . Then, increase R to have (3.7), itwill depend on ϕ and Y . This completes the proof of (3.4) and thereby (3.1). We concludethat κ is trivial. (cid:3) In the more general case, for arbitrary k , we are less precise. Proposition 3.1.
There are exceptional representations θ GL n − k and θ G k such that θ C,ψ k isembedded in a finite direct sum of copies of the representation ϑ ⊗ ( θ G k ) C U k ,ψ k , ϑ = ⎧⎪⎪⎪⎨⎪⎪⎪⎩ θ GL n k = , ind ̃ GL n − k ̃ GL ◻ n − k (( θ GL n − k ) ◻ ) k > . Here ϑ ⊗ ( θ G k ) C U k ,ψ k is regarded as a representation of ̃ St ψ k ⋉ U n by extending it trivially on U n − k . The representations θ GL n − k and θ G k are related to θ via (2.7) and (2.8) . If k = ,the embedding is in fact an isomorphism and there is only one summand, otherwise thereare [ F ∗ ∶ F ∗ ] summands.Proof of Proposition 3.1. For n =
1, we have C = U and k =
0, whence θ C,ψ k = θ U and theresult follows immediately from (2.8).Assume n >
1. Further assume k < n /
2, otherwise there is nothing to prove. The main partof the proof is to show that U n − k acts trivially on θ C,ψ k . Of course, this holds for U n − k ∩ C .Let V k = U n − k ∩ U n and note that Z n − k, k = U n − k ∩ M n . Clearly U n − k = V k ⋊ Z n − k, k . Thefollowing claims imply that the action of U n − k is trivial. Claim 3.2. θ C,ψ k = θ V k C,ψ k . Claim 3.3. θ V k C,ψ k = θ U n − k C,ψ k . Before proving the claims, let us deduce the proposition. Clearly θ U n − k C,ψ k = ( θ U n − k ) C U k ,ψ k .Assume k >
0. Then by (2.7), θ U n − k C,ψ k ⊂ I ◻ ( θ GL n − k , θ G k ) C U k ,ψ k . (3.9)According to Lemma 2.2, I ◻ ( θ GL n − k , θ G k ) is the finite direct sum of [ F ∗ ∶ F ∗ ] copies ofind ̃ M n − k p − ( GL ◻ n − k × G k ) (( θ GL n − k ) ◻ ⊗ θ G k ) . Let St ′ ψ k be the stabilizer of ψ k in M k , when ψ k is regarded as a character of C U k and M k < G k < G n . The double coset spaceGL ◻ n − k × G k / M n − k / GL n − k × St ′ ψ k is trivial. Then by virtue of the Geometric Lemma of Bernstein and Zelevinsky [BZ77](Theorem 5.2),ind ̃ M n − k p − ( GL ◻ n − k × G k ) (( θ GL n − k ) ◻ ⊗ θ G k ) C U k ,ψ k = ind p − ( GL n − k × St ′ ψk ) p − ( GL ◻ n − k × St ′ ψk ) (( θ GL n − k ) ◻ ⊗ ( θ G k ) C U k ,ψ k ) . Equality (2.1) implies the subgroups ̃ GL n − k and ̃ Sp k commute. Thereforeind p − ( GL n − k × St ′ ψk ) p − ( GL ◻ n − k × St ′ ψk ) (( θ GL n − k ) ◻ ⊗ ( θ G k ) C U k ,ψ k ) = ( ind ̃ GL n − k ̃ GL ◻ n − k ( θ GL n − k ) ◻ ) ⊗ ( θ G k ) C U k ,ψ k . Thus I ◻ ( θ GL n − k , θ G k ) C U k ,ψ k is the direct sum of representations ϑ ⊗ ( θ G k ) C U k ,ψ k . Theproposition follows from this. Note that for k = θ U n − k C,ψ k = θ U n and we can apply (2.8)instead of (2.7), then (3.9) becomes θ U n C = θ GL n ⊗ θ G . Proof of Claim 3.2.
A character ψ ⋆ k of V k C extending ψ k is defined by its restriction to thenontrivial coordinates on the ( n + ) -th column of v ∈ V k . We call ψ ⋆ k nontrivial if thisrestriction is nontrivial. We prove that the Jacquet module of θ C,ψ k with respect to V k and ψ ⋆ vanishes for any nontrivial ψ ⋆ . The group GL n − k < St ψ k acts transitively on thesecharacters. Therefore, it is enough to show ( θ C,ψ k ) V k ,ψ ⋆ k = θ V k C,ψ ⋆ k = , ψ ⋆ k ( v ) = ψ ( v ,n + ) , v ∈ V k . This follows immediately from Theorem 4, because any character of U extending ψ ⋆ k ∣ U hasa nonzero length. Thus θ C,ψ k = θ V k C,ψ k . (cid:3) Proof of Claim 3.3.
For k = V = U n ) . Assume k >
0. The claimfollows once we show that for any nontrivial character µ of Z n − k, k , ( θ V k C,ψ k ) Z n − k, k ,µ = . (3.10)The group Z n − k, k is abelian and GL n − k × Sp k acts on the characters of Z n − k, k . Write anelement z ∈ Z n − k, k in the form z = z ( z , z , z , z ) = ⎛⎜⎜⎜⎝ I n − k − z z z z I k − ⎞⎟⎟⎟⎠ . We may assume that µ does not depend on the coordinates of z and z , and depends on z .For simplicity, also assume µ ( z ( , , z , )) = ψ ( z ) .We use the local analog of “exchanging roots”, proved by Ginzburg, Rallis and Soudry[GRS99a] (Lemma 2.2). (For the global setting see [Gin90, GRS01, Sou05, GRS11].) Let Z < Z n − k, k be the subgroup of elements z ( z , , , ) and Z , , < Z n − k, k be the subgroupconsisting of elements z ( , z , z , z ) . Clearly Z n − k, k = Z ⋅ Z , , (a direct product). Alsoconsider the subgroup E = ⎧⎪⎪⎪⎨⎪⎪⎪⎩⎛⎜⎝ I n − k − e I k ⎞⎟⎠⎫⎪⎪⎪⎬⎪⎪⎪⎭ . In general, if π is a smooth representation of Z n − k, k ⋊ E , then by [GRS99a] (Lemma 2.2),as Z , , -modules π Z n − k, k ,µ = π Z , , ⋊ E,µ . Indeed, it is simple to check that the list of properties stated in the lemma are satisfied inthis setting (in the notation of [GRS99a], C = Z , , , X = Z and Y = E ). Remark 3.1.
Lemma 2.2 of [GRS99a] was stated for unipotent subgroups of symplecticgroups, but the arguments are general and hold in our setting. See also Section 2.3 of [GRS99a] . It follows that as Z , , -modules ( θ V k C,ψ k ) Z n − k, k ,µ = θ ( V k C ) ⋊ ( Z , , E ) ,ψ k µ . Conjugating the right-hand side by a Weyl element of G n , whose action on N n is given bythe action of diag (( I k + I n − k − ) , , ( I n − k − I k + )) , we obtain that θ ( V k C ) ⋊ ( Z , , E ) ,ψ k µ is a quotient of ( θ U ,ψ ) U ′ ,ψ . Here ψ ( u ) = ψ ( u , ) , U ′ is a certain subgroup of U (obtained from the conjugation of C ) and ψ ( u ) = ψ ( u , n − ) . Note that ψ corresponds to µ and the coordinate z while ψ corresponds to the character ψ k of C and the ( n − k + , n + k ) -th coordinate of c ∈ C . Thecharacter ψ is nontrivial on U ′ . Finally, by Proposition 4 of Bump, Friedberg and Ginzburg[BFG06] (which is easily extended to G n , given the analog of Theorem 4 in [Kap14a]), θ U ,ψ is a quotient of θ U (this is valid for n ≥
3, here 0 < k < n / n ≥ U actstrivially on θ U ,ψ and therefore ( θ U ,ψ ) U ′ ,ψ = (cid:3)(cid:3) Distinguished representations
Let G be either GL n or G n . Let τ be an admissible representation of G with a centralcharacter ω τ . Assume that θ and θ ′ are a pair of exceptional representations of ̃ G . We saythat τ is ( θ, θ ′ ) -distinguished if Hom G ( θ ⊗ θ ′ , τ ∨ ) ≠ . The following result describes the upper hereditary property of a distinguished representationof GL n , when induced to a representation of G n . Following the notation of Section 2.8,we denote the exceptional representation of ̃ G corresponding to χ and ψ by θ Gχ,ψ . For anyrepresentation σ of GL n , s ∈ C and a character µ of F ∗ , one forms a representation σ ∣ det ∣ s ⊗ µ of M n . Put I ( σ, s, µ ) = Ind G n Q n ( δ / Q n σ ∣ det ∣ s ⊗ µ ) . Proposition 4.1.
Let τ be a ( θ GL n χ,ψ , θ GL n χ ′ ,ψ ′ ) -distinguished representation of GL n and set η = ( χχ ′ ) − . Then I ( τ, / , η ) is a ( θ G n χ,ψ , θ G n χ ′ ,ψ ′ ) -distinguished representation of G n .Proof of Proposition 4.1. By definition the spaceTri GL n ( τ, θ GL n χ,ψ , θ GL n χ ′ ,ψ ′ ) of GL n -equivariant trilinear forms on τ × θ GL n χ,ψ × θ GL n χ ′ ,ψ ′ is nonzero. ThereforeTri M n ( τ ⊗ η, θ GL n χ,ψ ⊗ θ G χ, , θ GL n χ ′ ,ψ ′ ⊗ θ G χ ′ , ) ≠ . (4.1)According to (2.8), ( θ G n χ,ψ ) U n = θ GL n ∣ ⋅ ∣ ( n − )/ χ,ψ ⊗ θ G χ, . Applying Frobenius reciprocity we see that θ G n χ,ψ is a subrepresentation ofInd ̃ G n ̃ Q n ( δ n − n Q n θ GL n χ,ψ ⊗ θ G χ, ) . (4.2)A similar result holds for θ G n χ ′ ,ψ ′ .We define T ∈ Tri G n ( I ( τ, / , η ) , θ G n χ,ψ , θ G n χ ′ ,ψ ′ ) and prove it is nonzero. Let ϕ belong to the space θ G n χ,ψ , regarded as an element of (4.2), andsimilarly let ϕ ′ belong to the space of θ G n χ ′ ,ψ ′ . Also take f in the space of I ( τ, / , η ) . Now if L ≠ L ( f ( q ) , ϕ ( q ) , ϕ ′ ( q )) = δ Q n ( q ) L ( f ( ) , ϕ ( ) , ϕ ′ ( )) , q ∈ Q n . Thus the following integral is (formally) well defined (see e.g. [BZ76] 1.21), T ( f, ϕ, ϕ ′ ) = ∫ Q n / G n L ( f ( g ) , ϕ ( g ) , ϕ ′ ( g )) dg. It is absolutely convergent according to the Iwasawa decomposition. Since T satisfies thenecessary equivariance properties, it remains to show T ≠
0. Assume L ( x, y, y ′ ) ≠ f supported on Q n N , for a small compact open neighborhood N of theidentity in G n , and such that f (( a, b ) uv ) = δ / Q n ( a )∣ det a ∣ / η ( b ) τ ( a ) x, ∀ ( a, b ) ∈ GL n × G , u ∈ U n , v ∈ N . (4.3)We may assume ϕ ( ) = y (because θ GL n χ,ψ ⊗ θ G χ, is irreducible) and ϕ ′ ( ) = y ′ . Using theIwasawa decomposition and Q n N ∩ K = ( Q n ∩ K ) N then yields T ( f, ϕ, ϕ ′ ) = ∫ ( Q n ∩ K )N L ( f ( k ) , ϕ ( k ) , ϕ ′ ( k )) dk. Since L is invariant with respect to Q n ∩ K , taking a sufficiently small N (with respect to ϕ and ϕ ′ ), the dk -integration reduces to a nonzero constant multiple of L ( f ( ) , ϕ ( ) , ϕ ′ ( )) ,which is nonzero. We conclude that I ( τ, / , η ) is ( θ G n χ,ψ , θ G n χ ′ ,ψ ′ ) -distinguished. (cid:3) Let τ be a representation of G as above. Write θ = θ χ,ψ and θ ′ = θ χ ′ ,ψ ′ . Since θ χ,ψ = χθ ,ψ ,we may assume χ = χ ′ =
1, perhaps twisting τ by a character. For simplicity, we then saythat τ is ( ψ, ψ ′ ) -distinguished.If n is even, the characters ψ and ψ ′ can be ignored, because θ ,ψ does not depend on ψ . If n is odd and τ is ( ψ , ψ ′ ) -distinguished, then for any ψ there is ψ ′ such that τ is ( ψ, ψ ′ ) -distinguished. Indeed, write ψ ( x ) = ψ ( αx ) for some α ∈ F ∗ and put ψ ′ ( x ) = ψ ′ ( αx ) ,then by Claim 2.5 and its proof, θ ,ψ ⊗ θ ,ψ ′ = θ ,ψ ⊗ θ ,ψ ′ .In light of these observations, we say that τ is distinguished if for any ψ there is ψ ′ suchthat τ is ( ψ, ψ ′ ) -distinguished. Proposition 4.1 implies, Corollary 4.2.
Let τ be a distinguished representation of GL n . Then I ( τ, / , ) is distin-guished. Now we prove Theorem 1. Namely, for any pair θ and θ ′ of exceptional representations of ̃ G n , and a generic character ψ of N n , ( θ ⊗ θ ′ ) N n ,ψ = . We consider the filtrations of θ and θ ′ corresponding to the Jacquet functor along C = C U n . The kernel of this functor is glued from representations induced from the Jacquetmodules described in Section 3. Taking the twisted Jacquet functor along N n truncatessome of these quotients and, essentially, reduces the problem to a representation inducedfrom ( θ G k ) C U k ,ψ k ⊗ ( θ ′ G k ) C U k ,ψ − k . Theorem 3 then enables us to further reduce the problem,to the vanishing of ind GL k Sp k U n ( ω ψ ⊗ ω ψ − ) N n ,ψ , which essentially follows from the results of Offenand Sayag on Klyachko models ([OS08], see also [Kly83]). Proof of Theorem 1.
By an analog of the Geometric Lemma of Bernstein and Zelevinsky([BZ77] Theorem 5.2 and [BZ76] 5.9-5.12), as a ̃ Q n -module, θ is glued fromind ̃ Q n ̃ St ψk U n ( θ C,ψ k ) , ≤ k ≤ ⌊ n / ⌋ . (See Section 3 for the notation.) Then as a Q n -module θ ⊗ θ ′ is glued fromind ̃ Q n ̃ St ψk U n ( θ C,ψ k ) ⊗ ind ̃ Q n ̃ St ψk ′ U n ( θ ′ C,ψ − k ′ ) , ≤ k, k ′ ≤ ⌊ n / ⌋ . (4.4)We prove that the Jacquet functor with respect to N n and ψ vanishes on each of theserepresentations.Since functions in ind ̃ Q n ̃ St ψk U n ( θ C,ψ k ) are compactly supported modulo ̃ St ψ k U n , and C isnormal in Q n , by Lemma 2.1, ( ind ̃ Q n ̃ St ψk U n ( θ C,ψ k ) ⊗ ind ̃ Q n ̃ St ψk ′ U n ( θ ′ C,ψ − k ′ )) N n ,ψ = ⎧⎪⎪⎨⎪⎪⎩( ind ̃ Q n ̃ St ψk U n ( θ C,ψ k ⊗ θ ′ C,ψ − k )) N n ,ψ k = k ′ , k ≠ k ′ . (4.5)To see this consider f in the space of ind ̃ Q n ̃ St ψk U n ( θ C,ψ k ) and f ′ in the space of ind ̃ Q n ̃ St ψk ′ U n ( θ ′ C,ψ − k ′ ) ,and look at the Jacquet-Langlands integral ∫ C c ⋅ ( f ⊗ f ′ )( g, g ′ ) dc = ∫ C f ( gc ) f ′ ( g ′ c ) dc = f ( g ) f ′ ( g ′ ) ∫ C ψ k ( g c ) ψ − k ′ ( g ′ c ) dc, where C < C is a compact subgroup.Since θ C,ψ k ⊗ θ ′ C,ψ − k is a non-genuine representation of ̃ St ψ k , we can replace the representa-tion on the right-hand side of (4.5) with ( ind Q n St ψk U n ( θ C,ψ k ⊗ θ ′ C,ψ − k )) N n ,ψ . Define ϑ with respect to θ GL n − k as in Proposition 3.1 and similarly, define ϑ ′ with re-spect to θ ′ GL n − k . According to the proposition, θ C,ψ k is embedded in a finite direct sum ofrepresentations ϑ ⊗ ( θ G k ) C U k ,ψ k , which are trivial on U n − k . PutΠ k = ϑ ⊗ ϑ ′ ⊗ ( θ G k ) C U k ,ψ k ⊗ ( θ ′ G k ) C U k ,ψ − k . It is enough to prove that for all 0 ≤ k ≤ ⌊ n / ⌋ , ( ind Q n St ψk U n Π k ) N n ,ψ = . (4.6)This holds for k =
0, simply because U n is normal in Q n , Π is trivial on U n while ψ is not.The case of k = n / Claim 4.3.
Equality (4.6) holds for k = n / . Lastly, assume 0 < k < n /
2. Set Q = Q n − k ∩ Q n . By transitivity of induction ( ind Q n St ψk U n Π k ) N n ,ψ = ( ind Q n Q ( ind Q St ψk U n Π k )) N n ,ψ . The representation ind Q St ψk U n Π k is trivial on U n − k . By virtue of the Geometric Lemmaof Bernstein and Zelevinsky ([BZ77] Theorem 5.2), the representation on the right-handside is glued from Jacquet modules of ind Q St ψk U n Π k . Note that in general, the quotientsare representations induced from Jacquet modules, here the induction is trivial because thestabilizer of the character ψ of N n is N n × G .Let W be a set of representatives to the double cosets Q / Q n /( N n G ) . That is, Q n =∐ w ∈W Qw − N n G . We can take the elements w to be Weyl elements of GL n . When ψ ∣ w U n − k ∩ N n ≠
1, the quotient corresponding to w vanishes. This implies there is only one quotient, corre-sponding to w = ( I k I n − k ) , namely δ ⋅ ( ind Q St ψk U n Π k ) N GL n − k × N k ,ψ . Here δ is some modulus character, hereby ignored, and N GL n − k × N k < M n − k . As a G -module, this representation is isomorphic to ( ϑ ⊗ ϑ ′ ) N GL n − k ,ψ ⊗ ind Q k St ψk U k (( θ G k ) C U k ,ψ k ⊗ ( θ ′ G k ) C U k ,ψ − k ) N k ,ψ . Here ψ is regarded as a generic character of N GL n − k and N k . Since the case k = n / Q k St ψk U k (( θ G k ) C U k ,ψ k ⊗ ( θ ′ G k ) C U k ,ψ − k ) N k ,ψ = , Equality (4.6) follows.
Proof of Claim 4.3.
For k = n /
2, Π k = θ C,ψ k ⊗ θ ′ C,ψ − k . Now we apply Theorem 3. For simplicityof computations, we can replace ψ k with the character defined in the proof of the theorem,then use the epimorphism ℓ ∶ ̃ St ψ k ⋉ U n → ̃ Sp k ⋉ H n given there. By Theorem 3, θ C,ψ k isisomorphic to the direct sum of copies of ω ψ . Pull ω ψ back to a representation of ̃ St ψ k ⋉ U n .Note that the G part of St ψ k was ignored in the proof of Theorem 3, since it acts by acharacter, so we can ignore this here as well. Equality (4.6) will follow from ( ind Q n St ψk U n ( ω ψ ⊗ ω ψ − )) N n ,ψ = . (4.7)We need some notation. Put π = ω ψ ⊗ ω ψ − , G = P ○ n, , V = Z n, , P = Sp k ⋉ V, X = P / G. Note that C / Q n ≅ G = GL n ⋉ V (in fact, C / Q n ≅ G × G but G was ignored), this isomorphismrestricts to an isomorphism C / U n ≅ V . In this manner ψ is also a character of V , ψ ( v ) = ψ ( v n,n + ) . Since π is trivial on C , we can regard it as a representation of P and if π = ind GP ( π ) , ind Q n St ψk U n ( π ) ≅ π as G -modules.We apply the theory of l -sheafs of Bernstein and Zelevinsky ([BZ76] 1.13 and Section 6).In the following, we freely use their terminology and definitions. Let ( X, F ) be the l -sheaf corresponding to π ([BZ76] 2.23). The group N n acts on X by u ⋅ x = xu − and on F by u ⋅ ϕ ( x ) = ψ − ( u ) ϕ ( u − ⋅ x ) . An N n -invariant F -distribution on X is an element ofHom N n ( π, ψ ) . Since Hom N n ( π, ψ ) is the algebraic dual of π N,ψ , to prove (4.7) we will showthat there are no nonzero N n -invariant F -distributions on X .The action of N n on X is constructive ([BZ76] Theorem A). If x ∈ X , let P x = x − P ∩ N n be the stabilizer of x in N n . The orbit of x is N n ⋅ x . The mapping u ⋅ x ↦ ( P x ) u − inducesa homeomorphism N n ⋅ x ≅ P x / N n ([BZ76] 1.6). The restriction of F to the orbit of x (thisrestriction is an l -sheaf, because the action is constructive) is isomorphic to ind N n P x ( x − π ) ,where x − π is the representation of P x acting in the space of π by x − π ( z ) = π ( x z ) .By virtue of Theorem 6.9 of [BZ76], to show there are no nonzero N n -invariant F -distributions on X , it is enough to prove that for each representative x ,Hom N n ( ind N n P x ( x − π ) , ψ ) = Hom P x ( x − π , ψ ) = . We can take representatives x ∈ P ○ n − , < GL n . Then P x = Sp xk ⋉ V , where Sp xk = x − Sp k ∩ N GL n .In addition, because P ○ n − , fixes ψ ∣ V , ( x − π ) V,ψ = x − (( π ) V,ψ ) . HenceHom P x ( x − π , ψ ) = Hom Sp xk ( x − (( π ) V,ψ ) , ψ ) . Note that ( π ) V,ψ = ( ω ψ ⊗ ω ψ − ) H n ,µ for the nontrivial µ given in Claim 2.4. According tothat claim ( π ) V,ψ is the trivial one-dimensional representation of ℓ − ( Sp k ∩ P ○ n − , ) . Since x N GL n < P ○ n − , for any x ∈ P ○ n − , , x − (( π ) V,ψ ) is trivial on Sp xk (the epimorphism ℓ is easilyseen to be harmless here). But Offen and Sayag ([OS08] Proposition 2, we use H r,r ′ with r = r ′ = n , in their notation) proved that ψ ∣ Sp xk ≠ x ∈ GL n . This impliesHom Sp xk ( x − (( π ) V,ψ ) , ψ ) =
0, as required. (cid:3)(cid:3)
Corollary 4.4.
Let τ be an irreducible unitary supercuspidal ( θ GL n χ,ψ , θ GL n χ ′ ,ψ ′ ) -distinguishedrepresentation of GL n . Then the Langlands quotient of I ( τ, / , ( χχ ′ ) − ) is ( θ G n χ,ψ , θ G n χ ′ ,ψ ′ ) -distinguished. In particular, if τ is distinguished, so is the Langlands quotient of I ( τ, / , ) .Proof of Corollary 4.4. According to Proposition 4.1,Hom G n ( θ G n χ,ψ ⊗ θ G n χ ′ ,ψ ′ , I ( τ, / , ( χχ ′ ) − ) ∨ ) ≠ . Theorem 1 implies I ( τ, / , ( χχ ′ ) − ) is reducible (because τ is generic), then by Cassel-man and Shahidi [CS98] (Theorem 1), the Langlands quotient LQ ( I ( τ, / , ( χχ ′ ) − )) is non-generic, and the unique irreducible subspace of I ( τ, / , ( χχ ′ ) − ) is generic. Also note that thelength of I ( τ, / , ( χχ ′ ) − ) is 2 (because it is reducible and τ is supercuspidal, see [BZ77] 2.8).Now the result follows from Theorem 1 and the left exactness of the Hom functor. (cid:3) As a corollary, we can now prove Theorem 2. Namely, for an irreducible unitary supercus-pidal τ , being distinguished is equivalent to the occurrence of a pole at s = L ( s, τ, Sym ) . Proof of Theorem 2. If τ is distinguished, as in the proof of Corollary 4.4 we see thatI ( τ, / , ) is reducible, then according to Casselman and Shahidi [CS98] (Proposition 5.3,their Conjecture 1.1 was proved for G n in [Asg02]) L ( s, τ, Sym ) has a pole at s = L ( s, τ, Sym ) has a pole at s =
0. As an application ofthe descent method of Ginzburg, Rallis and Soudry (see e.g., [GRS97b, GRS99a, GRS99b,GRS01, GSS02, JS03, JS04, Sou05, Sou06, GRS11]), one can globalize τ to a cuspidal auto-morphic representation π of GL n ( A ) , such that L S ( s, π, Sym ) has a pole at s = τ is distinguished ([BG92]Theorem 7.6). (cid:3) Corollary 4.5. If τ is an irreducible unitary supercuspidal distinguished representation, itmust be self-dual. Remark 4.1.
The analogous result for an irreducible supercuspidal ( θ GL n χ,ψ , θ GL n χ ′ ,ψ ′ ) -distinguished τ should also hold. One needs to verify the applicability of the globalization argument (see [HS12] ). Given exceptional representations θ and θ ′ of ̃ G (as in the beginning of this section), wecan consider the space of θ ⊗ θ ′ as a model for representations of G . We refer to the dimensionof Hom G ( θ ⊗ θ ′ , τ ∨ ) as the multiplicity of τ . The next proposition relates the multiplicitiesof τ and I ( τ, / , η ) . In the case of GL n , Kable [Kab01] conjectured that the multiplicity of an irreduciblerepresentation is at most one. He proved this for n ≤
3, and for arbitrary n under a certainhomogeneity condition ([Kab01] Corollary 6.1). It is reasonable to believe multiplicity onealso holds in the context of G n (see [LM14a] Remark 4.2). Proposition 4.6.
Let τ be an irreducible tempered representation of GL n , put η = ( χχ ′ ) − and assume ∣ η ∣ = . Then dim Hom G n ( θ G n χ,ψ ⊗ θ G n χ ′ ,ψ ′ , I ( τ, / , η ) ∨ ) = dim Hom GL n ( θ GL n χ,ψ ⊗ θ GL n χ ′ ,ψ ′ , τ ∨ ) . (4.8) In particular, the representation τ is ( θ GL n χ,ψ , θ GL n χ ′ ,ψ ′ ) -distinguished if and only if I ( τ, / , η ) is ( θ G n χ,ψ , θ G n χ ′ ,ψ ′ ) -distinguished. Remark 4.2. If τ is ( θ GL n χ,ψ , θ GL n χ ′ ,ψ ′ ) -distinguished, in particular ω τ ( z ⋅ I n ) = η ( z n ) for all z ∈ F ∗ ( ω τ - the central character of τ ). It follows that ∣ η ∣ = , because a tempered representationis unitary. In this case by (4.8) τ enjoys multiplicity one if and only if I ( τ, / , η ) does.Proof of Proposition 4.6. Set θ = θ GL n χ,ψ , θ = θ G n χ,ψ and similarly θ ′ and θ ′ (with χ ′ and ψ ′ ).For brevity, put d = ∣ det ∣ . Applying Frobenius reciprocity,Hom G n ( θ ⊗ θ ′ , I ( τ, / , η ) ∨ ) = Hom GL n (( θ ⊗ θ ′ ) U n ∣ GL n , d ( n − )/ τ ∨ ) . Using the notation of Section 3, the representation ( θ ⊗ θ ′ ) U n is filtered by representations W k = ( ind Q n St ψk U n ( θ C,ψ k ⊗ θ ′ C,ψ − k )) U n , ≤ k ≤ ⌊ n / ⌋ . The representation W is a quotient of ( θ ⊗ θ ′ ) U n and the kernel of the mapping ( θ ⊗ θ ′ ) U n → W is filtered by W k with 1 ≤ k ≤ ⌊ n / ⌋ . Regard W k as a representation of GL n (by restrictionfrom M n ).According to Proposition 3.1, W = d ( n − )/ θ ⊗ θ ′ . ThenHom GL n ( W , d ( n − )/ τ ∨ ) = Hom GL n ( θ ⊗ θ ′ , τ ∨ ) . Thus (4.8) will follow, once we prove that for k > GL n ( W k , d ( n − )/ τ ∨ ) = . (4.9)If n is even, we claim the following. Claim 4.7.
For any irreducible generic representation σ of GL n , Hom GL n ( W n / , σ ) = . Inparticular Equality (4.9) holds for k = n / . The proof will be given below.Assume 0 < k < n /
2. The representation θ C,ψ k is embedded in a finite direct sum ofrepresentations ϑ ⊗ ( θ G k ) C U k ,ψ k (Proposition 3.1). The representation ϑ is semisimple,and by Theorem 3 the representation ( θ G k ) C U k ,ψ k is also semisimple. Hence θ C,ψ k ⊗ θ ′ C,ψ − k is embedded in a semisimple representation, which is the finite direct sum of Π k (Π k = ϑ ⊗ ϑ ′ ⊗ ( θ G k ) C U k ,ψ k ⊗ ( θ ′ G k ) C U k ,ψ − k ). Therefore, using the exactness of induction andJacquet functors, W k is a quotient of ( ind Q n St ψk U n ( ⊕ Π k )) U n . We conclude that we may replace W k in (4.9) with ( ind Q n St ψk U n Π k ) U n . An application of [BZ77] (Theorem 5.2) yields ( ind Q n St ψk U n Π k ) U n = ind GL n P n − k, k ( ϑ ⊗ ϑ ′ ⊗ W ′ ) , where W ′ = ( ind Q k St ψk U k ( θ G k C U k ,ψ k ⊗ θ ′ G k C U k ,ψ − k )) U k ∣ GL k . Note that the relevant double coset space (for [BZ77] Theorem 5.2) is ( Q n − k ∩ Q n )/ Q n / Q n ,there is only one representative to consider.Hence to deduce (4.9), it is enough to proveHom GL n ( ind GL n P n − k, k ( ϑ ⊗ ϑ ′ ⊗ W ′ ) , d ( n − )/ τ ∨ ) = . The left-hand side equalsHom GL n ( d ( − n )/ τ, ind GL n P n − k, k ( δ P n − k, k ( ϑ ⊗ ϑ ′ ⊗ W ′ ) ∨ )) = Hom P n − k, k ( d ( − n )/ τ Z n − k, k , δ P n − k, k ( ϑ ⊗ ϑ ′ ⊗ W ′ ) ∨ ) = Hom P n − k, k ( d ( − n )/ δ − / P n − k, k ϑ ⊗ ϑ ′ ⊗ W ′ , j ( τ ) ∨ ) . (4.10)Here j ( τ ) = δ − / P n − k, k τ Z n − k, k (the normalized Jacquet functor). It suffices to show that thelast space (4.10) vanishes, when j ( τ ) is replaced by any of its irreducible subquotients. Let τ ⊗ τ be one such subquotient. Then (4.10) vanishes if eitherHom GL k ( W ′ , d ( k − )/ τ ∨ ) or(4.11) Hom GL n − k ( d ( − n − k )/ ϑ ⊗ ϑ ′ , τ ∨ ) (4.12)are zero. As explained in the proof of Proposition 4.1 of Lapid and Mao [LM14a], since τ is tempered, either τ is generic, or the central character ω τ of τ satisfies ∣ ω τ ∣ = ∣ det ∣ α for some α >
0. In the former case (4.11) vanishes by Claim 4.7. In the latter case (4.12)vanishes. Indeed, z ⋅ I n − k acts on ϑ ⊗ ϑ ′ by d ( − + n + k )/ η − ( z ( n − k ) ) and since ∣ η ∣ =
1, thisaction is unitary on d ( − n − k )/ ϑ ⊗ ϑ ′ , but ω τ is not unitary. Proof of Claim 4.7.
Put k = n / π = ind Q n St ψk U n ( θ C,ψ k ⊗ θ ′ C,ψ − k ) . We need to proveHom GL n ( π U n , σ ) = . We will show that π U n has a filtration, whose quotients are all isomorphic to ind GL n Sp n /
1. Thensince σ is irreducible and generic, Hom GL n ( ind GL n Sp n / , σ ) = H ,n )and the result follows.We turn to prove the filtration of π U n . As in the proof of Claim 4.3, Theorem 3 impliesthat π U n is filtered by copies of the representation ( ind Q n St ψk U n ( ω ψ ⊗ ω ψ − )) U n . We prove ( ind Q n St ψk U n ( ω ψ ⊗ ω ψ − )) U n = ind GL n Sp n / . (4.13)(Recall that the left-hand side is regarded as a representation of GL n .)Since U n is normal in Q n , Lemma 2.1 implies ( ind Q n St ψk U n ( ω ψ ⊗ ω ψ − )) U n = ind Q n St ψk U n (( ω ψ ⊗ ω ψ − ) H n ) . In more detail, if f belongs to the space of ind Q n St ψk U n ( ω ψ ⊗ ω ψ − ) , the Jacquet-Langlandsintegral takes the form ∫ U f ( gu ) du = ∫ U ( ω ψ ⊗ ω ψ − )( g u ) f ( g ) du, for a compact subgroup U < U n . Because f is compactly supported modulo St ψ k U n , thisintegral vanishes for all g ∈ Q n if and only if f ( g ) belongs to the space of ( ω ψ ⊗ ω ψ − )( H n ) for all g . It remains to use the exactness of induction.According to Claim 2.4, ( ω ψ ⊗ ω ψ − ) H n is the trivial one-dimensional representation of Sp k and (4.13) holds. (cid:3)(cid:3) We can now improve Corollary 4.2 for tempered representations:
Corollary 4.8.
Let τ be an irreducible tempered representation of GL n . Then τ is distin-guished if and only if I ( τ, / , ) is distinguished. The small representation of SO n + Bump, Friedberg and Ginzburg [BFG03] constructed and studied the small representa-tions for SO n + . In this section we briefly recall their results and formulate our results forSO n + . The cover ̃ SO n + was obtained by restricting the 4-fold cover of SL n + of Mat-sumoto [Mat69]. It is a “double cover” in the sense that the square of the cocycle is trivialon the kernel of the spinor norm.We use the same notation of G n , e.g., B n = B SO n + (see Section 2.3), T n is the diagonaltorus and Q k is a standard maximal parabolic subgroup. If ( a, g ) ∈ GL k × SO ( n − k ) + , ( a, g ) is embedded in M k as diag ( a, g, J kt a − J k ) . The block compatibility formula now reads (see[BFG03] (2.20)) σ SL n + (( a, g ) , ( a ′ , g ′ )) = σ SL k + ( diag ( a, det a − ) , diag ( a ′ , det a ′− )) ( det a, det a ′ ) σ SL ( n − k ) + ( g, g ′ ) . The benefit of this cover is that the preimages of GL k and SO ( n − k ) + commute, thus thetensor can be used to describe representations of Levi subgroups. Restriction of the cover toGL k is a double cover.The small representation θ = θ SO n + is unique; it is the representation of ̃ SO n + corre-sponding to the exceptional character ξ ( s ( diag ( t , . . . , t n , , t − n , . . . , t − ))) = n ∏ i = ∣ t i ∣ n − i + . According to [BFG03] (Theorem 2.3), θ U k = θ GL k ⊗ θ SO ( n − k ) + , where θ GL k was explicitlygiven, and by [BFG03] (Theorem 2.6) and [BFG06] (Proposition 3), θ U ,ψ ( ) = ψ ( ) is nonzero (using the notation of Section 2.8).Theorems 1 and 3 remain valid as stated. Proposition 3.1 now takes the form θ C,ψ k = θ GL n − k ⊗ ( θ SO k + ) C U k ,ψ k . Here θ GL n − k is uniquely determined. Indeed, this equality replaces (3.9) because θ U n − k is atensor of representations.Regarding Proposition 4.1, assume τ is ( ψ, ψ ′ ) -distinguished. Twisting τ by some squaretrivial character, we obtain a ( ψ, ψ ) -distinguished representation. Then, perhaps after usinganother twist of τ , it becomes ( θ GL n , θ GL n ) -distinguished for the exceptional representation θ GL n such that θ U n = θ GL n . Now the proof of the proposition proceeds as in the case of G n . So,in this case one must start with a ( θ GL n , θ GL n ) -distinguished representation of GL n , in order to obtain a distinguished representation I ( τ, / , ) of SO n + . Corollary 4.4 is applicable for τ such that Proposition 4.1 is valid. Proposition 4.6 remains valid. References [AS06] M. Asgari and F. Shahidi. Generic transfer for general spin groups.
Duke Math. J. , 132(1):137–190,2006.[AS11] M. Asgari and F. Shahidi. Functoriality for general spin groups, 2011. arXiv:1101.3467 [math.NT],http://arxiv.org/abs/1101.3467.[Asg00] M. Asgari.
On holomorphy of local Langlands L-functions . ProQuest LLC, Ann Arbor, MI, 2000.Thesis (Ph.D.)–Purdue University.[Asg02] M. Asgari. Local L -functions for split spinor groups. Canad. J. Math. , 54(4):673–693, 2002.[BFG00] D. Bump, S. Friedberg, and D. Ginzburg. A Rankin-Selberg integral using the automorphicminimal representation of SO ( ) . J. Ramanujan Math. Soc. , 15(2):81–124, 2000.[BFG03] D. Bump, S. Friedberg, and D. Ginzburg. Small representations for odd orthogonal groups.
In-ternat. Math. Res. Notices , (25):1363–1393, 2003.[BFG06] D. Bump, S. Friedberg, and D. Ginzburg. Lifting automorphic representations on the doublecovers of orthogonal groups.
Duke Math. J. , 131(2):363–396, 2006.[BG92] D. Bump and D. Ginzburg. Symmetric square L -functions on GL ( r ) . Ann. of Math. (2) ,136(1):137–205, 1992.[BK94] R. Brylinski and B. Kostant. Minimal representations of E , E , and E and the generalizedCapelli identity. Proc. Nat. Acad. Sci. U.S.A. , 91(7):2469–2472, 1994.[BLS99] W. D. Banks, J. Levy, and M. R. Sepanski. Block-compatible metaplectic cocycles.
J. ReineAngew. Math. , 507:131–163, 1999.[BS98] R. Berndt and R. Schmidt.
Elements of the representation theory of the Jacobi group , volume 163of
Progress in Mathematics . Birkh¨auser Verlag, Basel, 1998.[BZ76] I. N. Bernstein and A. V. Zelevinsky. Representations of the group GL ( n, F ) where F is a localnon-Archimedean field. Russian Math. Surveys , 31(3):1–68, 1976.[BZ77] I. N. Bernstein and A. V. Zelevinsky. Induced representations of reductive p -adic groups I. Ann.Scient. ´Ec. Norm. Sup. , 10(4):441–472, 1977.[Car93] R. W. Carter.
Finite groups of Lie type . Wiley Classics Library. John Wiley & Sons Ltd., Chich-ester, 1993. Conjugacy classes and complex characters, Reprint of the 1985 original, A Wiley-Interscience Publication.[CM93] D. H. Collingwood and W. M. McGovern.
Nilpotent orbits in semisimple Lie algebras . Van Nos-trand Reinhold Mathematics Series. Van Nostrand Reinhold Co., New York, 1993.[CS98] W. Casselman and F. Shahidi. On irreducibility of standard modules for generic representations.
Ann. Scient. ´Ec. Norm. Sup. , 31(4):561–589, 1998.[FK86] Y. Z. Flicker and D. A. Kazhdan. Metaplectic correspondence.
Inst. Hautes ´Etudes Sci. Publ.Math. , 64:53–110, 1986.[Gin90] D. Ginzburg. L -functions for SO n × GL k . J. Reine Angew. Math. , 405:156–180, 1990.[Gin06] D. Ginzburg. Certain conjectures relating unipotent orbits to automorphic representations.
IsraelJ. Math. , 151:323–355, 2006.[GJS10] D. Ginzburg, D. Jiang, and D. Soudry. Periods of automorphic forms, poles of L -functions andfunctorial lifting. Sci. China Math. , 53(9):2215–2238, 2010.[GJS11] D. Ginzburg, D. Jiang, and D. Soudry. On certain automorphic descents to GL . Internat. Math.Res. Notices , 2011(21):4779–4820, 2011.[GRS97a] D. Ginzburg, S. Rallis, and D. Soudry. On the automorphic theta representation for simply lacedgroups.
Israel J. Math. , 100:61–116, 1997.[GRS97b] D. Ginzburg, S. Rallis, and D. Soudry. Self-dual automorphic GL n modules and construction ofa backward lifting from GL n . Internat. Math. Res. Notices. , 687(14):687–701, 1997.[GRS99a] D. Ginzburg, S. Rallis, and D. Soudry. On a correspondence between cuspidal representations of GL n and ̃ Sp n . J. Amer. Math. Soc. , 12(3):849–907, 1999.[GRS99b] D. Ginzburg, S. Rallis, and D. Soudry. On explicit lifts of cusp forms from GL m to classicalgroups. Ann. of Math. (2) , 150(3):807–866, 1999. [GRS01] D. Ginzburg, S. Rallis, and D. Soudry. Generic automorphic forms on SO ( n + ) : functorial liftto GL ( n ) , endoscopy, and base change. Internat. Math. Res. Notices. , 2001(14):729–764, 2001.[GRS03] D. Ginzburg, S. Rallis, and D. Soudry. On Fourier coefficients of automorphic forms of symplecticgroups.
Manuscripta Math. , 111(1):1–16, 2003.[GRS11] D. Ginzburg, S. Rallis, and D. Soudry.
The Descent Map from Automorphic Representations ofGL(n) to Classical Groups . World Scientific Publishing, Singapore, 2011.[GS05] W. T. Gan and G. Savin. On minimal representations definitions and properties.
Represent.Theory , 9:46–93 (electronic), 2005.[GSS02] D. Ginzburg, Rallis S, and D. Soudry. Endoscopic representations of ̃ Sp n . J. Inst. Math. Jussieu ,1(1):77–123, 2002.[HS12] J. Hundley and E. Sayag. Descent construction for GSpin groups, 2012. arXiv:1110.6788v2[math.NT], http://arxiv.org/pdf/1110.6788.pdf.[Ike94] T. Ikeda. On the theory of Jacobi forms and Fourier-Jacobi coefficients of Eisenstein series.
J.Math. Kyoto Univ. , 34(3):615–636, 1994.[JR92] H. Jacquet and S. Rallis. Symplectic periods.
J. Reine Angew. Math. , 423:175–197, 1992.[JR96] H. Jacquet and S. Rallis. Uniqueness of linear periods.
Compositio Math. , 102(1):65–123, 1996.[JS03] D. Jiang and D. Soudry. The local converse theorem for SO ( n + ) and applications. Ann. ofMath. , 157(3):743–806, 2003.[JS04] D. Jiang and D. Soudry. Generic representations and local Langlands reciprocity law for p -adic SO ( n + ) . In H. Hida, D. Ramakrishnan, and F. Shahidi, editors, Contributions to automor-phic forms, geometry, and number theory: a volume in honor of Joseph Shalika , pages 457–519,Baltimore, MD, 2004. Johns Hopkins Univ. Press.[Kab01] A. C. Kable. The tensor product of exceptional representations on the general linear group.
Ann.Sci. ´Ecole Norm. Sup. (4) , 34(5):741–769, 2001.[Kab02] A. C. Kable. On a conjecture of Savin.
Internat. Math. Res. Notices. , 2002(30):1601–1627, 2002.[Kap] E. Kaplan. The theta period of a cuspidal automorphic representation of GL ( n ) . 2014,doi:10.1093/imrn/rnt358, to appear in Internat. Math. Res. Notices.[Kap14a] E. Kaplan. The double cover of odd general spin groups, small representations and applications.arXiv:1411.4082 [math.RT], http://arxiv.org/pdf/1411.4082.pdf, 2014.[Kap14b] E. Kaplan. Representations distinguished by pairs of exceptional representations and a conjectureof Savin. arXiv:1411.3697 [math.RT], http://arxiv.org/pdf/1411.3697.pdf, 2014.[Kaz90] D. Kazhdan. The minimal representation of D . In Operator algebras, unitary representations,enveloping algebras, and invariant theory (Paris, 1989) , volume 92 of
Progr. Math. , pages 125–158. Birkh¨auser Boston, Boston, MA, 1990.[Kly83] A. A. Klyachko. Models for complex representations of groups GL ( n, q ) . Mat. Sb. (N.S.) ,120(162)(3):371–386, 1983.[KP84] D. A. Kazhdan and S. J. Patterson. Metaplectic forms.
Inst. Hautes ´Etudes Sci. Publ. Math. ,59:35–142, 1984.[KPW02] D. Kazhdan, B. Pioline, and A. Waldron. Minimal representations, spherical vectors and excep-tional theta series.
Comm. Math. Phys. , 226(1):1–40, 2002.[KS90] D. Kazhdan and G. Savin. The smallest representation of simply laced groups. In
Festschriftin honor of I. I. Piatetski-Shapiro on the occasion of his sixtieth birthday, Part I (Ramat Aviv,1989) , volume 2 of
Israel Math. Conf. Proc. , pages 209–223. Weizmann, Jerusalem, 1990.[LM13] E. Lapid and Z. Mao. Whittaker-Fourier coefficients of cusp forms on ̃ Sp n : reduction to a localstatement, 2013. arXiv:1401.0198 [math.NT], http://arxiv.org/pdf/1401.0198v3.pdf.[LM14a] E. Lapid and Z. Mao. Model transition for representations of metaplectic type, 2014. to appearin Internat. Math. Res. Notices.[LM14b] E. Lapid and Z. Mao. On an analogue of the Ichino–Ikeda conjecture for Whittaker coefficients onthe metaplectic group, 2014. arXiv:1404.2905 [math.NT], http://arxiv.org/pdf/1404.2905v2.pdf.[LM15] E. Lapid and Z. Mao. A conjecture on Whittaker-Fourier coefficients of cusp forms. J. NumberTheory , 146:448–505, 2015.[LS08] H. Y. Loke and G. Savin. On minimal representations of Chevalley groups of type D n , E n and G . Math. Ann. , 340(1):195–208, 2008. [LS10] H. Y. Loke and G. Savin. Modular forms on non-linear double covers of algebraic groups.
Trans.Amer. Math. Soc. , 362(9):4901–4920, 2010.[Mat69] H. Matsumoto. Sur les sous-groupes arithm´etiques des groupes semi-simples d´eploy´es.
Ann. Sci.´Ecole Norm. Sup. (4) , 2(1):1–62, 1969.[Mat09] I. Mati´c. Levi subgroups of p -adic Spin ( n + ) . Math. Commun. , 14(2):223–233, 2009.[Mez04] P. Mezo. Metaplectic tensor products for irreducible representations.
Pacific J. Math. , 215(1):85–96, 2004.[Mui04] G. Mui´c. Reducibility of standard representations for classical p -adic groups. In Functional anal-ysis VIII (Dubrovnik, 2003) , volume 47 of
Various Publ. Ser. , pages 132–145. Univ. Aarhus,Aarhus, 2004.[MVW87] C. Mœglin, M.-F. Vign´eras, and J.-L. Waldspurger.
Correspondances de Howe sur un corps p -adique , volume 1291 of Lecture Notes in Mathematics . Springer-Verlag, Berlin, 1987.[OS08] O. Offen and E. Sayag. Uniqueness and disjointness of Klyachko models.
J. Funct. Anal. ,254(11):2846–2865, 2008.[Per81] P. Perrin. Repr´esentations de Schr¨odinger, indice de Maslov et groupe metaplectique. In
NonCommutative Harmonic Analysis and Lie Groups (Proc. Marseille-Luminy 1980) , volume 880 of
Lecture Notes in Math. , pages 370–407. Springer-Verlag, 1981.[PPS89] S. J. Patterson and I. I. Piatetski-Shapiro. The symmetric-square L -function attached to a cuspidalautomorphic representation of GL . Math. Ann. , 283(4):551–572, 1989.[PR12] D. Prasad and D. Ramakrishnan. Self-dual representations of division algebras and Weil groups:a contrast.
Amer. J. Math. , 134(3):749–767, 2012.[Pra93] D. Prasad. Weil representation, Howe duality, and the theta correspondence. In
Theta functions:from the classical to the modern , volume 1 of
CRM Proc. Lecture Notes , pages 105–127. Amer.Math. Soc., Providence, RI, 1993.[Rao93] R. Rao. On some explicit formulas in the theory of Weil representations.
Pacific J. Math. ,157(2):335–371, 1993.[Ros96] J. R. Roskies.
The minimal representation of SO(4,3) over a P-adic field . Stanford University,California, 1996.[Sab96] H. Sabourin. Une repr´esentation unipotente associ´ee `a l’orbite minimale: le cas de SO ( , ) . J.Funct. Anal. , 137(2):394–465, 1996.[Sav92] G. Savin. On the tensor product of theta representations of GL . Pacific J. Math. , 154(2):369–380,1992.[Sav93] G. Savin. An analogue of the Weil representation for G . J. Reine Angew. Math. , 434:115–126,1993.[Sav94] G. Savin. Dual pair G J × PGL , G J is the automorphism group of the Jordan algebra J . Invent.Math. , 118(1):141–160, 1994.[Sha92] F. Shahidi. Twisted endoscopy and reducibility of induced representations for p -adic groups. DukeMath. J. , 66(1):1–41, 1992.[Sou05] D. Soudry. On Langlands functoriality from classical groups to GL n . In J. Tilouine, H. Carayol,M. Harris, and M.-F. Vign´eras, editors, Formes Automorphes (I); Actes du Semestre du CEB,Printemps 2000 , volume 298 of
Ast´erisque , pages 335–390, 2005.[Sou06] D. Soudry. Rankin–Selberg integrals, the descent method, and Langlands functoriality. In
ICM
Duke Math. J. , 163(1):175–266,2014.[Tor97] P. Torasso. M´ethode des orbites de Kirillov-Duflo et repr´esentations minimales des groupes simplessur un corps local de caract´eristique nulle.
Duke Math. J. , 90(2):261–377, 1997.[vD78] G. van Dijk. Smooth and admissible representations of p -adic unipotent groups. Compositio Math. ,37(1):77–101, 1978. [Vog81] D. A. Vogan, Jr. Singular unitary representations. In Noncommutative harmonic analysis and Liegroups (Marseille, 1980) , volume 880 of
Lecture Notes in Math. , pages 506–535. Springer, Berlin,1981.[Wei64] A. Weil. Sur certains groupes d’op´erateurs unitaires.
Acta Math. , 111(1):143–211, 1964.
E-mail address ::