Classical Theta Lifts for Higher Metaplectic Covering Group
aa r X i v : . [ m a t h . R T ] A ug CLASSICAL THETA LIFTS FOR HIGHER METAPLECTIC COVERINGGROUPS
SOLOMON FRIEDBERG AND DAVID GINZBURG
Abstract.
The classical theta correspondence establishes a relationship between auto-morphic representations on special orthogonal groups and automorphic representations onsymplectic groups or their double covers. This correspondence is achieved by using as in-tegral kernel a theta series on the metaplectic double cover of a symplectic group that isconstructed from the Weil representation. There is also an analogous local correspondence.In this work we present an extension of the classical theta correspondence to higher degreemetaplectic covers of symplectic and special orthogonal groups. The key issue here is thatfor higher degree covers there is no analogue of the Weil representation, and additional in-gredients are needed. Our work reflects a broader paradigm: constructions in automorphicforms that work for algebraic groups or their double covers should often extend to higherdegree metaplectic covers. Introduction
Theta series provide a way to construct correspondences between spaces of automorphicforms. For example, suppose that G = SO ( V ) and G = Sp ( V ) are orthogonal andsymplectic groups, resp. Then these groups embed in a symplectic group G := Sp ( V ⊗ V ).Let F be a number field and A its ring of adeles. There is a family of theta functions θ φ onthe adelic metaplectic double cover M p ( V ⊗ V )( A ) (depending on some additional data φ ),and these functions may be used to create a correspondence between automorphic forms f on G ( A ) and automorphic forms f on G ( A ) or its double cover: f ( g , φ ) = Z θ φ ( g · g ) f ( g ) dg , where the integral is over the adelic quotient G ( F ) \ G ( A ). This correspondence, which ina low-rank case can be used to recreate the Shimura correspondence, has been studied bymany authors over the past half century (see for example Rallis [Ra2] and the referencestherein). There is also a local correspondence of irreducible smooth representations, theHowe correspondence, that is obtained by restricting the Weil representation to the imageof G · G in G (see Howe [Ho]), that has likewise received a great deal of attention. Thegoal of this paper is to extend these constructions to higher metaplectic covers of the groups G , G (beyond the double cover M p ) and to initiate an analysis of the resulting map ofrepresentations.
Date : August 29, 2020.2010
Mathematics Subject Classification.
Primary 11F27; Secondary 11F70.
Key words and phrases.
Metaplectic cover, theta lifting, Weil representation, unipotent orbit.This work was supported by the BSF, grant number 2016000, and by the NSF, grant number DMS-1801497(Friedberg). he double cover of the symplectic group, M p , was introduced by Weil [We] in his treat-ment of theta series. Over local or global fields F with enough roots of unity, there are higherdegree covers of classical groups as well, related to the work of Bass-Milnor-Serre on the con-gruence subgroup problem. These were first treated for simply connected algebraic groupsover F whose F -points are simple, simply connected, split over F and of rank at least two byMatsumoto [Mat], and in the context of number theory and automorphic forms for GL n byKubota [Ku] (when n = 2) and Kahzdan-Patterson [K-P] (for general n ). The constructionof covering groups was extended to a wide class of groups K -theoretically by Brylinski andDeligne [B-D]. Fan Gao [Gao1] gave a thorough treatment of metaplectic covers over localand global fields using the Brylinski-Deligne construction, and generalized Langlands’s workon the constant term of Eisenstein series to Eisenstein series on covering groups. Though formore general groups there are sometimes a number of inequivalent covers of a given degree,for the groups we treat here the covers are essentially unique, and locally may be regardedas restrictions of the covers of Matsumoto. We will describe the covers in detail below.In this paper, we construct a global theta lifting taking automorphic representations oncovers of a symplectic group to automorphic forms on covers of an orthogonal group. Herethe degrees of the covers must be compatible. The construction is via a theta kernel. Thedifficulty in this construction is that there is no known analogue of the Weil representationfor general degree covers. The classical theta functions may also be obtained as residues ofEisenstein series on M p ( A ), as shown by Ikeda [Ik1] (this may be regarded as an instanceof the Siegel-Weil formula; see Ikeda [Ik2]), and so a first attempt would be to mimic theconstruction above: e f ( g ) = Z e θ ( g · g ) e f ( g ) dg , where now e θ is an automorphic function realized as a residue of an Eisenstein series. Thisis indeed useful in many cases, including the lift of Bump and the authors [B-F-G2] for thedouble cover of an orthogonal group and the recent work of Leslie [L], which constructsCAP representations of the four-fold cover of a symplectic group using such a theta kernel.However, it is not sufficient to produce the lifting here. The reason is that this residue istoo large a representation, that is, it is not attached to the minimal nontrivial coadjointorbit or some other orbit of small Gelfand-Kirillov dimension. Thus it is too big to give acorrespondence by mimicking the standard procedure (i.e., restricting to the tensor productembedding).Instead we consider the product of two different theta functions—one coming from aresidue of an Eisenstein series, and the other coming from the Weil representation— anduse a Fourier coefficient of this product as integral kernel (given precisely in (3.3) below).In the case of the trivial cover of the orthogonal group, one of the theta functions and theFourier coefficient integral become trivial, and our construction reduces to the classical thetakernel construction. Hence the construction presented here may be viewed as an extensionof the classical theta integral construction. There is also a local analogue, and we show thatfor principal series induced from unramified quasicharacters in general position the liftingin the equal rank case gives a correspondence that is indeed compatible with the expectedmap of L -groups (these are discussed below). For the classical theta correspondence, localfunctoriality was established (in greater generality) by Rallis [Ra1], Section 6. o explain the L-group formalism that is related to our construction, let G denote one ofthe groups Sp l or SO k . Let r denote a positive integer, and G ( r ) ( A ) the r -fold metaplecticcovering group of G ( A ). For this group to be defined we need to assume that the field F contains all r -th roots of unity; for convenience we shall assume that F contains the 2 r -throots of unity. The notion of an L-group for metaplectic covering groups was developed ingeneral by Weissman [W], following a description in the split case by McNamara [McN].The assumption that F contains the 2 r -th roots avoids certain complications in Weissman’sconstruction (and is required by McNamara). In particular, though the Langlands dual groupfor a cover of an even orthogonal group goes as usual: ( SO ( r )2 a ) ∨ = SO a ( C ), the Langlandsdual group for covers of odd orthogonal and symplectic groups depends on the parity of r :( Sp ( r )2 l ) ∨ = ( SO l +1 ( C ) if r is odd Sp l ( C ) if r is even ( SO ( r )2 a +1 ) ∨ = ( Sp a ( C ) if r is odd SO a +1 ( C ) if r is even.As in the classical case, one then expects functorial liftings of automorphic representationsbetween the metaplectic covering groups of symplectic and orthogonal groups. However,as one can see from these dual groups, the groups involved in such a theta correspondenceshould sometimes be different than in the classical case. Specifically, if r is odd, then forsuitable a, l one expects a correspondence between automorphic representations of Sp ( r )2 l ( A )and SO ( r )2 a ( A ), and between automorphic representations of Sp (2 r )2 l ( A ) and SO ( r )2 a +1 ( A ). For r = 1, these maps are the classical theta correspondence. In this paper we introduce aconstruction which gives them for any odd r .Residues of Eisenstein series, when square-integrable, contribute to the residual spectrum,and these residues are frequently not generic. Indeed, the residue we consider generates anautomorphic representation that should have many vanishing classes of Fourier coefficients,and this property is key to establishing our results. However, the question of determining themaximal unipotent orbit that supports a non-zero Fourier coefficient, or locally a nonzerotwisted Jacquet module, is rather delicate, and has only recently been addressed for thegeneral linear group (Cai [C1]; see also Leslie [L]). We make a conjecture about this exactorbit (Conjecture 1), establish the vanishing of Fourier coefficients that it would imply,and establish a number of partial results towards the full conjecture, including proving it incertain cases. Though we do not resolve it in full, these are sufficient for the results presentedhere. See Section 4 below.The construction we present here can most likely be taken farther. For example, one canconsider the inverse correspondence from covers of orthogonal groups to covers of symplec-tic groups obtained by the same integral kernel, and in the local case one may formulateanalogues of the Howe Conjecture. One can also ask for the first non-zero occurrence of thelift (for the classical theta lift, see Roberts [Ro]), and this will be treated in a subsequentpaper [F-G5]. In particular, we will show that for given r , if Conjecture 1 below holds,then the integrals presented here do not vanish on a full theta tower, and also that a genericcuspidal genuine automorphic representation of Sp ( r )2 n ( A ) always lifts nontrivially to a genericgenuine representation on SO ( r )2 n + r +1 ( A ). Since Conjecture 1 is established here for r = 3 , r even, but this will require additional information about the unipotent orbits of thetaresidues. ur work suggests a broader paradigm: constructions in the theory of automorphic formsshould generalize to covers. We note three examples. First, as mentioned above, Fan Gao[Gao1] has extended Langlands’s work on the constant term of Eisenstein series to covers.Second, the doubling integrals of the authors, Cai and Kaplan [CFGK1] may be extendedto covers, as loosely sketched in our announcement [CFGK2] and worked out in detail byKaplan in [Kap]. See also Cai [C2]. Third, the work here indicates that the classical thetacorrespondence also generalizes. As noted above, Weissman [W] has defined a metaplectic L -group. Our suggestion is that not only the formalism of functoriality but also the integralsthat give L -functions or correspondences should often generalize. Of course, doing so mustinvolve new ideas, as is the case here. As another example, it is not straightforward to extendLanglands-Shahidi theory to covers as this theory uses the uniqueness of the Whittakermodel, and such uniqueness does not hold, even locally, for most covering groups. However,it should be possible to generalize this theory by replacing the Whittaker model for a cuspidalautomorphic representation τ on a metaplectic covering group with the Whittaker model forthe Speh representation attached to τ . Indeed, this Speh representation, a residue of anEisenstein series (and defined only conjecturally at the moment), is expected to have aunique Whittaker model by a generalization of Suzuki’s conjecture [Su1], [Su2]. (Suzuki’sconjecture concerns covers of GL n and we expect a similar phenomenon in general.)To conclude this introduction we describe the algebraic and representation theoretic struc-tures that are behind our construction. A classical reductive dual pair is a pair of subgroups( G , G ) inside a symplectic group Sp ( W ) which are mutual centralizers and which act re-ductively on W . Our algebraic structure is this:(1) Groups G , G , and two symplectic vector spaces W , W , with monomorphisms ι : G × G → Sp ( W ) , ι : G × G → Sp ( W ) , such that(a) via the map ι , ( G , G ) is a reductive dual pair in Sp ( W )(b) the images under ι of G × × G in Sp ( W ) commute, though they neednot be mutual centralizers;(2) A unipotent group U ⊂ Sp ( W ) which is normalized by ι ( G , G );(3) A character ψ U : U → C such that ι ( G , G ), acting by conjugation, stabilizes ψ U ,and the index of ι ( G , G ) in the full stabilizer of ψ U is finite;(4) A homomorphism l : U → H ( W ), where H ( W ) denotes the Heisenberg groupattached to W .In this paper, we take ( G , G ) = ( SO k , Sp n ). The map ι is the tensor product embeddinginto Sp nk , just as in the classical theta correspondence, while remaining ingredients aregiven in Sections 2 and 3 below. We actually consider covers, and extend the maps ι , ι tocovers there.To explain the representation theory that is employed, let ω ψ denote the Weil represen-tation of H ( W ) ⋊ M p ( W ) with additive character ψ , over either A or a completion of F .As noted above, the global theta correspondence (resp. locally, the Howe Correspondence)is based on restricting theta series constructed from ω ψ (resp. the representation ω ψ ) to theimage of a reductive dual pair ι ( G , G ). In our structure, we have two symplectic groups,and we restrict the tensor product of two small representations. Specifically, we form therepresentation ω ψ ⊗ Θ ( r )2 , where Θ ( r )2 is a theta representation of Sp ( r ) ( W ). We then restrict his to M p ( W ) × Sp ( r ) ( W ) where the groups act on the two representations via ι and ι ,resp. Globally, in making the integral kernel we also take a Fourier coefficient with respectto U and ψ U , where the action of U on the first factor is via ℓ ; see (3.3) below. The localmap, conjecturally a correspondence, makes use of a twisted Jacquet module with respectto U , where in defining this Jacquet module, the action of U on the first factor is again via ℓ . That is, if we write ( V , ω ψ ) and ( V , Θ ( r )2 ) as the representations over a local field, then J U,ψ U ( ω ψ ⊗ Θ ( r )2 ) = ( V ⊗ V ) /V with V = < ω ψ ( ℓ ( u )) v ⊗ Θ ( r )2 ( u ) v − v ⊗ ψ U ( u ) v | v ∈ V , v ∈ V , u ∈ U > .
It is this twisted Jacquet module that is restricted to G × G (with suitable covers, andonce again using the embeddings ι , ι ).Section 2 introduces the notation to be used in the sequel and treats the necessary foun-dations concerning metaplectic groups in some detail. In particular, we shall have occasionto use isomorphic but different covers of groups (these arise by modifying a 2-cocycle by acoboundary), and these are described precisely. Then Section 3 presents the integral (3.3)that describes the theta lifting. As indicated above, this integral requires a theta functionon a higher odd degree cover of a symplectic group Sp l and also a second theta function onthe double cover of a symplectic group, which is constructed using the Weil representation.The associated higher theta representation Θ ( r )2 l is analyzed in Section 4. In Conjecture 1we describe the expected unipotent orbit behavior of this representation, and in Theorem 1we give a proof of the Conjecture in certain cases and a partial proof in others. Section 5concerns cuspidality. In Theorem 2, we show that an analogue of the Rallis theta tower isvalid for this new class of theta lifts. The proof uses in an essential way both the propertiesof the higher theta representation Θ ( r )2 l and the description of the classical theta function onthe double cover. Section 6, the final section of this work, concerns the unramified lift. InTheorem 3, this is shown to be functorial for characters in general position in the equal rankcase. 2. Groups and Covering Groups
Let F be a number field with ring of adeles A . If H is any algebraic group defined over F , we write [ H ] for the quotient H ( F ) \ H ( A ). In particular, [ G a ] = F \ A .We will work with the symplectic and special orthogonal groups defined over F . We realizethem as follows. For m ≥ J m be the m × m matrix with 1 on the anti-diagonal andzero elsewhere. Let Sp m denote the symplectic group Sp m = (cid:26) g ∈ GL m | t g (cid:18) J m − J m (cid:19) g = (cid:18) J m − J m (cid:19)(cid:27) and for k ≥ SO k denote the split special orthogonal group SO k = (cid:8) g ∈ GL k | t g J k g = J k (cid:9) . (The construction presented here can be extended to the non-split case but we shall not doso here.)Throughout the paper, we make use of two embeddings. First, let n ≥ k ≥
2, andlet ι : SO k × Sp n → Sp nk denote the usual tensor product embedding. In matrices, if h ∈ SO k ( A ) and g ∈ Sp n ( A ), then ι ((1 , g )) = diag( g, . . . , g ) where g appears k times, and (( h, h ij I n ). Second, let r ≥ ι : SO k × Sp n → Sp n + k ( r − be theembedding given by ι ( h, g ) = diag( h, . . . , h, g, h ∗ , . . . , h ∗ ), where h , h ∗ each appear ( r − / h ∗ is determined so that the matrix is symplectic.Let Mat a × b denote the algebraic group of all matrices of size a × b , and when a = b , writeMat a = Mat a × a . For l ≥
1, let H l +1 denote the Heisenberg group in 2 l + 1 variables. Thisis the group with elements ( X, Y, z ) where
X, Y ∈ Mat × l and z ∈ Mat , and multiplicationgiven by( X , Y , z )( X , Y , z ) = ( X + X , Y + Y , z + z + ( X J l t Y − Y J l t X )) . The Heisenberg group H l +1 is isomorphic to a subgroup of Sp l +2 by the map τ ( X, Y, z ) = X Y zI l Y ∗ I l X ∗ , where the starred entries are uniquely determined by the requirement that the matrix besymplectic ( X ∗ = − J l t X , Y ∗ = J l t Y ). Also, let Mat a = { Z ∈ Mat a | t ZJ a = J a Z } , andlet Mat a = { Z ∈ Mat a | t ZJ a = − J a Z } .Next we define some parabolic and unipotent groups that will be of use. All parabolicsubgroups here are standard parabolic subgroups whose unipotent radical consists of uppertriangular unipotent matrices, and all of the unipotent groups we introduce are groups ofupper triangular unipotent matrices.For non-negative integers a, b and c , let P a,b,c denote the parabolic subgroup of Sp ab + c ) whose Levi part is GL a × . . . × GL a × Sp c where GL a appears b times. Let U a,b,c denote theunipotent radical of the group P a,b,c . Let 1 ≤ i ≤ b − α = a ( i − β = 2( ab + c ) − a ( i + 1),and set(2.1) u ia,b,c ( X ) = I α I a XI a I β I a X ∗ I a I α , X ∈ Mat a . The set of all matrices { u ia,b,c ( X ) | X ∈ Mat a } is a subgroup of U a,b,c which we denote by U ia,b,c . Given u ∈ U a,b,c , one may write it uniquely as u = u ia,b,c ( X i ) u ′ where X i ∈ Mat a and u ′ ∈ U a,b,c is such that all its ( p, j ) entries are zero when α + 1 ≤ p ≤ α + a and α + a + 1 ≤ j ≤ α + 2 a . We call u ia,b,c ( X i ) the i -th coordinate of u .Another subgroup of U a,b,c is the group U ′ a,c that consists of all matrices(2.2) u ′ a,c ( Y, Z ) = I a ( b − I a Y ZI c Y ∗ I a I a ( b − , Y ∈ Mat a × c , Z ∈ Mat a . imilarly, we define the u ′ a,c ( Y, Z ) coordinate of u . Then every u ∈ U a,b,c has a factorization(2.3) u = u ′ a,c ( Y, Z ) b − Y i =1 u ia,b,c ( X i ) u , where u ∈ U a,b,c has entries zero in the first b a × a blocks directly above the main diagonaland in positions Y and Z in (2.2) above (and so also zero in the corresponding last b superdiagonal blocks and in Y ∗ ). Let U a,b,c denote the subgroup of U a,b,c consisting of allmatrices (2.3) such that Y = 0. Also, let U a,b,c denote the subgroup of U a,b,c consisting of allmatrices (2.3) such that Y = Z = 0. Note that U a,b,c is a subgroup of U a,b,c .The group U ′ a,c has a structure of a generalized Heisenberg group. Define a homomorphism l : U a,b,c → H ac +1 as follows. Let y i ∈ Mat × c denote the i -th row of Y , and define(2.4) l ( u ′ a,c ( Y, Z )) = ( y a , y a − , . . . , y , tr( T k Z )) , where T k is the identity matrix I k if k is even, and T k = diag( I [ k/ , , I [ k/ ) if k is odd. (Notethat the center of U a,c consists of all matrices (2.2) such that Y = 0, and this is mapped tothe center of H ac +1 .) Then extend l trivially from U ′ a,c to U a,b,c .Also, when we work with Weyl groups, we shall always work with representatives whichhave one non-zero entry in each row and column whose value is ±
1; we call such matricesWeyl group elements. For the symplectic group Sp a , such a matrix is determined uniquelyby its entries in the first a rows and we sometimes specify it in this way (e.g. the proof ofProposition 5).We now set the notation for metaplectic groups. Fix an integer m ≥ F contains a full set of m -th roots of unity µ m . (Below, we will take m to be an odd integer r or to be 2 r .) If G is either the F ν points (where ν is a place of F ) or the A points of alinear algebraic group (that will be SO or Sp ), then an m -fold covering group e G of G is atopological central extension of G by µ m . The group consists of pairs ( g, ǫ ), g ∈ G , ǫ ∈ µ m ,with multiplication described in terms a (Borel measurable) 2-cocycle σ ∈ Z ( G, µ m ):( g , ǫ )( g , ǫ ) = ( g g , ǫ ǫ σ ( g , g )) . After fixing an embedding of µ m into C × , such groups naturally give rise to central extensionsof G by C × by the same multiplication rule. If β : G → C × is a (Borel measurable) mapwith β ( e ) = 1, then multiplying σ by the 2-coboundary ( g , g ) β ( g ) β ( g ) /β ( g g ) givesan isomorphic group, and we also consider such groups. One can construct a global 2-cocyclefrom local cocycles at each place by taking the product, but only if almost all factors are 1on the adelic points of G .We begin with the local constructions that we will use. Fix r > K be a localfield containing a full set of r -th roots of unity and let ( , ) r ∈ µ r be the local Hilbert symbol.If Sp m ( K ) is any symplectic group, let Sp ( r )2 m ( K ) (or simply Sp ( r )2 m ) denote the r -fold coveringof Sp m ( K ) constructed by Matsumoto [Mat] using the Steinberg symbol corresponding to( , ) − r . Since Sp m is simple and simply connected, this is the unique topological r -foldcovering group of Sp m ( K ); see for example Moore [Mo]. Hence, the cocycle is unique, butonly up to a coboundary. For any integer m , let σ m ∈ Z ( SL m ( K ) , µ r ) denote the 2-cocycleon SL m ( K ) constructed by Banks, Levy and Sepanski [B-L-S], Section 2 (when we need toindicate the cover, we write this σ ( r ) m ). This cocycle enjoys a block compatibility property([B-L-S], Theorem 1) that will be of use. We shall realize Sp ( r )2 m by restricting the cocycle m ∈ Z ( SL m ( K ) , µ r ) to Sp m ( K ) × Sp m ( K ), as in Kaplan [Kap], Section 1.4. Doing so,the block compatibility implies that if g , g ∈ SO k ( K ), h , h ∈ Sp n ( K ), then(2.5) σ n + k ( r − ( ι ( h , g ) , ι ( h , g )) = σ k ( h , h ) r − σ n ( g , g ) . Since σ k take values in µ r , we have σ k ( h , h ) r − = σ k ( h , h ) − . We realize the r -fold cover SO ( r ) k ( K ) of the orthogonal group SO k ( K ) by restricting the cocycle σ − k to SO k ( K ) × SO k ( K ). Then (2.5) implies that the map ι extends to a homomorphism of the coveringgroups ι ( r )2 : SO ( r ) k ( K ) × Sp ( r )2 n ( K ) → Sp ( r )2 n + k ( r − ( K ), given by(2.6) ι ( r )2 ((( h, ǫ ) , ( g, ǫ ))) = ( ι ( h, g ) , ǫ ǫ ) . We will also make use of the double cover Sp (2)2 nk ( K ) (this cover is unique up to isomor-phism), and its pullback via the tensor product embedding ι . Recall that when k is even(resp. k is odd), the pair ( SO k , Sp n ) (resp. ( SO k , Sp (2)2 n )) forms a reductive dual pair inside Sp (2)2 nk . We recall the local theory and give a reference for the adelization; this constructionunderlies the classical theta correspondence.For each m ≥
1, we realize the group Sp (2)2 m ( K ) by a Leray cocycle e σ m , which takesvalues in (generally) the eighth roots of unity. (We will be specific in the next paragraph.)This cocycle may be regarded as arising from the classical Weil representation attached to afixed nontrivial additive character and realized in the Schr¨odinger model; see Kudla [Ku2],Theorem 3.1. The cocycle is cohomologous to a “Rao cocycle” with values in µ = {± } (see Rao [Rao]).Information about the behavior of the metaplectic double cover with respect to the tensorproduct embedding may be found in Kudla [Ku1], [Ku2]. We use a cocycle given in thoseworks, adjusting the notation to take into account some different normalizations. Kudlaconsiders vector spaces V, W over the local field K of dimensions k and 2 n , resp., equippedwith nondegenerate bilinear (resp. symplectic) forms, and defines W = V ⊗ K W . Thetensor product of the forms gives a symplectic form on W . He then defines j W , j V byusing the tensor product embedding SO ( V ) × Sp ( W ) → Sp ( W ) and restricting to the firstand second factors, resp. Let λ be the natural isomorphism from Sp nk ( K ) (defined above)to the group Sp ( W ) in Kudla’s work (i.e. moving one symplectic form to another), suchthat λ ( ι ( h, j W ( h ) and λ ( ι (1 , g )) = j V ( g ) for all h ∈ SO k ( K ), g ∈ Sp n ( K ). Let c Y be the Leray cocycle on Sp nk ( K ) in Kudla’s notation. Then we realize Sp (2)2 nk ( K ) bymeans of the cocycle e σ nk ( g , g ) = c Y ( λ ( g ) , λ ( g )) (for g , g ∈ Sp nk ( K )). For k = 1 thisdefines Sp (2)2 n ( K ), with Y the maximal isotropic subspace of W given on p. 18 of [Ku2], whichcorresponds to 2 n -vectors with first n entries zero in our normalization. Then it is shown in[Ku2], Ch. II, Proposition 3.2, that there is a map β k : Sp n ( K ) → S ⊂ C × such that themap ι (2)1 given by ι (2)1 : SO k ( K ) × Sp n ( K ) → Sp (2)2 nk ( K ) ι (2)1 ( h, g ) = ( ι ( h, g ) , β k ( g ) − ) if k is even ι (2)1 : SO k ( K ) × Sp (2)2 n ( K ) → Sp (2)2 nk ( K ) ι (2)1 ( h, ( g, ǫ )) = ( ι ( h, g ) , β k ( g ) − ǫ ) if k is oddis an isomorphism.We now turn to the global construction. Let m ≥ M = Sp m . Wewill construct the r -fold cover of M ( A ). (We will then choose m = 2 n + k ( r − F bea number field, and for each place ν let σ ν be the cocycle e σ ( r )2 m described above for M ( F ν ). hen at almost all places, the local cocycle σ ν may be adjusted by a coboundary η ν so thatthe new cocycle ρ ν satisfies ρ ν ( κ, κ ′ ) = 1 if κ, κ ′ are in the hyperspecial maximal compactgroup M ( O ν ), where O ν is the ring of integers of F ν . (This may not be true, and thereis no adjustment, at a finite number of places, namely the ramified places of F , the placesdividing 2 r and the archimedean places.)More precisely, for unramified places, let η ν : M ( F ν ) → S be a continuous map thatsatisfies(2.7) σ ν ( m, m ′ ) = η ν ( mm ′ ) η ν ( m ) − η ν ( m ′ ) − for all m, m ′ ∈ M ( O ν ) . This uniquely determines η ν on M ( O ν ), and it is then extended to M ( F ν ). See [Kap], Section1.5. We will choose η ν so that for all h ∈ SO k ( F ν ), g ∈ Sp n ( F ν ),(2.8) η ν ( ι ( h, η ν ( ι (1 , g )) = η ν ( ι ( h, g )) . Indeed, if h ∈ SO k ( O ν ), g ∈ Sp n ( O ν ), then (2.8) holds, since σ ν ( ι ( h, , ι (1 , g )) = 1 by(2.5), and then (2.8) follows from (2.7). We may then choose the extension of η ν to M ( F ν )so that (2.8) holds in general. Then introduce the cocycle ρ ν ( g, g ′ ) = η ν ( g ) η n ( g ′ ) η ν ( gg ′ ) σ ν ( g, g ′ ) g, g ′ ∈ M ( F ν ) , and the 2-cocycle ρ = Q v ρ v of M ( A ). This is well-defined since if g, g ′ ∈ M ( A ), thenat almost all places g, g ′ ∈ M ( O ν ) and ρ ν ( g, g ′ ) = 1. We shall realize the global r -foldmetaplectic cover M ( r ) ( A ) of M ( A ) as the central extension of M ( A ) by µ r with respectto this cocycle. For g ∈ M ( A ) such that η ( g v ) = 1 for almost all v , let η ( g ) = Q v η v ( g v );in particular, this function is defined for g ∈ M ( F ) (see Takeda [Tak], Proposition 1.8 andKaplan [Kap], Section 1.5). Then (as a consequence of Hilbert reciprocity) M ( F ) embeds in M ( r ) ( A ) by the homomorphism m ( m, η − ( m )). Since it will always be clear from contextwhether we are working in a matrix group or a covering group, we shall not introducea separate notation for the image of M ( F ); instead, we regard M ( F ) as a subgroup of M ( r ) ( A ) via this map. We may then form the automorphic quotient M ( F ) \ M ( r ) ( A ) andconsider genuine automorphic representations on this quotient.For our construction below, we will proceed as follows. Given n, k , let m = 2 n + k ( r − ρ on M ( A ). We then define 2-cocycles on SO k ( A ) and Sp n ( A )by pulling back ρ via the maps ι (( ⋆, ι ((1 , ⋆ )), resp. In view of (2.5), this gives r -foldcovers of these groups, which we write SO ( r ) k ( A ), Sp ( r )2 n ( A ), whose local components at ν areisomorphic to the local metaplectic r -fold covers constructed above (with K = F ν ). Also,using (2.8) it is not difficult to check that the map ι ( r )2 : SO ( r ) k ( A ) × Sp ( r )2 n ( A ) → Sp ( r )2 n + k ( r − ( A )given by (2.6) is a homomorphism.Similarly one can define a global two-fold cover Sp (2)2 nk ( A ) whose multiplication is given bya cocycle σ (2)2 nk which is the product of the cocycle e σ nk,ν above adjusted by a coboundary ateach place ν . This cocycle restricts to the trivial cocycle on ι ( SO k ( A ) , and to a cocyclewhich is either cohomologically trivial if k is even or nontrivial if k is odd on ι (1 , Sp n ( A )) .See for example Sweet [Sw], Sections 1.8 and 2.4. Thus there are homomorphisms ι (2)1 : SO k ( A ) × Sp n ( A ) → Sp (2)2 nk ( A ) ι (2)1 ( h, g ) = ( ι ( h, g ) , δ k ( g )) if k is even ι (2)1 : SO k ( A ) × Sp (2)2 n ( A ) → Sp (2)2 nk ( A ) ι (2)1 ( h, ( g, ǫ )) = ( ι ( h, g ) , δ k ( g ) ǫ ) if k is odd , here δ k : Sp n ( A ) → S is a Borel measurable map. We also introduce the r -fold cover f Sp ( r )2 n ( A ), that is obtained by multiplying the cocycle ρ by the coboundary δ k ( g , g ) := δ k ( g g ) δ − k ( g ) δ − k ( g ). The map i ( g, ǫ ) = ( g, δ k ( g ) ǫ ) is an isomorphism from Sp ( r )2 n ( A ) to f Sp ( r )2 n ( A ).We realize the 2 r -fold cover Sp (2 r )2 n ( A ) by using the 2-cocycle which is the product of the2-cocycles ρ and σ (2)2 n . For any a there is a canonical projection p (1) : Sp ( a )2 n ( A ) → Sp n ( A )given by projection onto the first factor. Then the 2 r -fold cover Sp (2 r )2 n ( A ) is isomorphic to thefibre product of Sp ( r )2 n ( A ) and Sp (2)2 n ( A ) over Sp n ( A ) with respect to these projections. Fixingsuch an isomorphism (equivalently, an isomorphism µ r ∼ = µ r × µ ), the group Sp (2 r )2 n ( A ) thuscomes equipped with projections p ( r ) : Sp (2 r )2 n ( A ) → Sp ( r )2 n ( A ), p (2) : Sp (2 r )2 n ( A ) → Sp (2)2 n ( A )that are homomorphisms. We also use p (1) for the projection from SO ( r ) k ( A ) → SO k ( A ).Last, for fixed k odd we introduce the group f Sp (2 r )2 n ( A ) by using the 2-cocycle ρσ (2)2 n δ k . Onceagain the map i ( g, ǫ ) = ( g, δ k ( g ) ǫ ) is an isomorphism from Sp (2 r )2 n ( A ) to f Sp (2 r )2 n ( A ).As noted above, the group Sp n ( F ) embeds in Sp ( κr )2 n ( A ) for κ = 1 ,
2. Though Sp ( κr )2 n isnot an algebraic group, we abuse the notation slightly and write [ Sp ( κr )2 n ] for the automorphicquotient Sp n ( F ) \ Sp ( κr )2 n ( A ). Also, δ k ( Sp n ( F )) = 1 (see Sweet [Sw], Proposition 2.4.2), so i induces a bijection of the automorphic quotients of Sp ( r )2 n ( A ) and f Sp ( r )2 n ( A ).We conclude this section by mentioning several additional groups of matrices that embedin the covering groups. First, for any local field F ν and the cover SL ( r ) d ( F ν ) described above,any upper unipotent subgroup N ( F ν ) of SL d ( F ν ) is canonically split by the trivial section u ( u, N ( A ) splits in SL ( r ) d ( A ) by means of a section of N ( A ), n ( n, η ( n ) − ).Moreover, this section is canonical (Mœglin and Waldspurger [M-W], Appendix 1). Anylower unipotent group is also split by a canonical section n − ( n − , η ( n − )) and thesesplittings are compatible with the action of the Weyl group (which embeds in the coveringgroup) by conjugation. That is, if n − is lower triangular and w n − := wn − w − is in N ( A ),then w ( n − , η ( n − )) = ( w n − , η ( w n − )). (See for example, [Kap], equation (1.11).) From nowon, we consider all unipotent groups as embedded in the relevant covering groups by suchsections (and once again do not introduce a separate notation). With this convention, allnotions related to unipotent orbits extend to covering groups without change.Similarly, working over F ν or A , for a = 0, let t ( a ) = diag( a, a − , . . . , a, a − ) ∈ Sp l ( l ≥ a , a ∈ F ∗ ν are r -th powers, then σ ( r )2 l,ν ( t ( a ,ν ) , t ( a ,ν )) = 1 (this follows fromthe well-known formula for the cocycle σ ( r )2 l,ν on diagonal matrices; see for example [B-L-S],Section 3). This equality implies that the map η ν restricted to the group { t ( a ) | a ∈ O ∗ ,rν } isa homomorphism. Using this observation and the explicit description of η ν on SL ( O ν ) (dueto Kubota), it follows that η ν ( t ( a ν )) = 1 for each place ν such that a ν ∈ O ∗ ,rν . Accordingly,the map t ( a ) e t ( a ) := ( t ( a ) , η − ( t ( a ))) is a well-defined embedding of { t ( a ) | A ∗ ,r } into Sp ( r )2 l ( A ). This embedding will appear in the proof of Proposition 6 below. . Description of the Integral Kernel
As mentioned in the introduction, the integral kernel we work with here requires twodifferent theta functions. The first is a function on the metaplectic double cover Sp (2)2 l ( A ) of Sp l ( A ). Let ψ denote a nontrivial character of F \ A . Let Θ (2)2 l denote the theta representationdefined on the group Sp (2)2 l ( A ) formed using ψ . We refer to [G-R-S1], Section 1, part 6, forthe definitions and the action of the Weil representation. This representation extends to thegroup H l +1 ( A ) · Sp (2)2 l ( A ). The theta kernel will involve a function θ (2) ,ψ l in Θ (2)2 l .To give the second, let r > F contains a full set of r -th roots of unity µ r . Fix an embedding ǫ : µ r → C × . Let Θ ( r )2 l bethe theta representation on the group Sp ( r )2 l ( A ) that is genuine with respect to ǫ ; that is, thefunctions θ ( r ) in Θ ( r )2 l transform by central character ǫ , i.e., θ ( r ) (( I l , µ ) g ) = ǫ ( µ ) θ ( r ) ( g ). Thisrepresentation is defined via residues of the minimal parabolic Eisenstein series on Sp ( r )2 l ( A ),similarly to the construction for GL r in [K-P]. For its definition and basic properties, see[F-G2, Section 2]; see also Gao [Gao2]. We will discuss the Fourier coefficients attached todifferent unipotent orbits for this representation in Section 4 below.To describe the global construction, we work with the groups U a,b,c defined just before(2.1) above. Define a character ψ U a,b,c of the group [ U a,b,c ] as follows. Given u ∈ U a,b,c , write u = u ′ a,c ( Y, Z ) Q i u ia,b,c ( X i ) u as in (2.3). Define(3.1) ψ U a,b,c ( u ) = ψ (tr( X + · · · + X b − )) . By restriction, this character is also a character of [ U a,b,c ]. We will be concerned with aspecific choice of the numbers a, b and c .Fix two integers k and n , which will index the sizes of the orthogonal and symplecticgroups, resp., as in Section 2. Let r = ( r − /
2, and set a = k, b = r and c = n . In ourmain construction, we will take a Fourier coefficient corresponding to the unipotent orbit(( r − k n ) of the group Sp n + k ( r − (see, for example, [G]). This is an integral over [ U k,r ,n ]against the character ψ U k,r ,n . It follows from [C-M] that the image of SO k × Sp n underthe embedding ι is in the stabilizer of this orbit and this Fourier coefficient, so the Fouriercoefficient gives rise to a function on this product. In view of our work with cocycles above,this extends to covering groups.We are now ready to discuss our global integrals. Fix k and let κ = 1 if k is even, and κ = 2 if k is odd. Let g ∈ Sp ( κr )2 n ( A ) and h ∈ SO ( r ) k ( A ). Then the Fourier coefficient ofinterest to us, corresponding to the unipotent orbit (( r − k n ), is(3.2) Z [ U k,r ,n ] θ (2) ,ψ nk ( l ( u ) ι (2)1 ( p (1) ( h ) , p ( κ ) ( g ))) θ ( r )2 n + k ( r − ( u ι ( r )2 ( h, p ( r ) ( g ))) ψ U k,r ,n ( u ) du. Here θ ( r )2 n + k ( r − is a vector in the space of the representation Θ ( r )2 n + k ( r − , θ (2) ,ψ nk is a vector inthe space of the representation Θ (2)2 nk (which depends on ψ ), and the map l is as in (2.4) with a = k , b = r and c = n . Note that both of these theta functions are genuine functions. Weshall use this Fourier coefficient as our integral kernel. o formulate the theta lift, let π ( κr ) denote a genuine irreducible cuspidal automorphicrepresentation of Sp ( κr )2 n ( A ) where the metaplectic groups are constructed in the prior sec-tion. For fixed k , by means of the isomorphism i we may realize π ( κr ) by means of genuineautomorphic functions on the group f Sp ( κr )2 n ( A ), and we do so henceforth. Here genuine meansthat the functions in π ( κr ) transform under the center of f Sp ( κr )2 n ( A ), { (1 , µ ) | µ ∈ µ κr } , bya fixed embedding ǫ ′ : µ κr → C × which is compatible with ǫ and, if k is odd, with theisomorphism of µ r with µ r × µ selected above. Specifically, let p denote the projectionfrom any covering group to its second factor (a root of unity of F ). Then we require that ǫ ′ ( µ ) = ǫ ( p ( p ( r ) (1 , µ ))) p ( p ( κ ) (1 , µ )) for all µ ∈ µ κr . Let ϕ ( κr ) denote a vector in the spaceof π ( κr ) . Definition 1.
The theta lift of π ( κr ) is the representation σ ( r ) n,k of SO ( r ) k ( A ) generated by thefunctions f ( h ) defined by (3.3) f ( h ) = Z [ Sp ( κr )2 n ] Z [ U k,r ,n ] ϕ ( κr ) ( i ( g )) θ (2) ,ψ nk ( l ( u ) ι (2)1 ( p (1) ( h ) , p ( κ ) ( g ))) θ ( r )2 n + k ( r − ( u ι ( r )2 ( h, p ( r ) ( g ))) ψ U k,r ,n ( u ) du dg. as each of ϕ ( κr ) , θ (2) ,ψ nk , θ ( r )2 n + k ( r − varies over the functions in its representation space. For convenience we sometimes shorten the notation in working with (3.3) below, writing ι ( h, g ) in place of ι (2)1 ( p (1) ( h ) , p ( κ ) ( g ))) and ι ( h, g ) in place of ι ( r )2 ( h, p ( r ) ( g ))).In (3.3) the covers in g are compatible (otherwise this integral would vanish identically fortrivial reasons). Also, the integral converges absolutely. This follows from the cuspidality ofthe representation π ( κr ) . Each function f is a genuine function on SO ( r ) k ( A ). The construc-tion given by the space generated by the integrals (3.3) defines a mapping from the set ofirreducible cuspidal representations of Sp ( κr )2 n ( A ) to the set of representations of the quotient SO k ( F ) \ SO ( r ) k ( A ). In the following Sections we will study the properties of this mapping.As mentioned in the introduction, this construction may be viewed as an extension ofthe well-known theta lift associated to the reductive dual pair ( SO k , Sp n ) if k is even, andto the reductive dual pair ( SO k , Sp (2)2 n ) if k is odd. Indeed, in integral (3.3) we assumedthat r ≥ r = 1, let Θ ( r )2 n + k ( r − be the trivial representation,and let U k, ,n be the trivial group, then we get the theta lift. Just as the theta lift can beused to go from either group in the reductive dual pair to the other, we could also use thesame integral kernel to construct a mapping from the set of irreducible cuspidal automorphicrepresentations of SO ( r ) k ( A ) to automorphic representations of Sp ( κr )2 n ( A ).There is also a local analogue of this integral given by a Hom space, which naturallygives rise to a map of representations over local fields that formally generalizes the Howecorrespondence. This will be treated for unramified principal series in Section 6 below.To conclude this section, let us mention that the theta kernel above is consistent withthe Dimension Equation of [G], [F-G4]. As explained in [F-G4], Section 6, the DimensionEquation states that when a theta kernel gives a correspondence, there is an equality re-lating the dimensions of the groups in the integral and the dimensions of the automorphicrepresentations appearing in and arising from the construction. Here the dimension of an utomorphic representation means its Gelfand-Kirillov dimension in the sense of [G]. In ourcase, this is the equalitydim( π ) + dim(Θ (2)2 nk ) + dim(Θ ( r )2 n + k ( r − ) = dim( Sp n ) + dim( U k,r ,n ) + dim( σ ) , where π , σ are as above. In the case that π and σ are both generic, and when n = [ k/ SO ( r ) k and Sp ( κr )2 n . And indeed, it may be checked(using formulas (2), (3) in [F-G4]) that in this situation the above equation holds, assumingConjecture 1 below. This equality is another motivation for choosing the orbit (( r − k n )for the unipotent integration in (3.2).4. The Unipotent Orbit of The Theta Representation
In this section we discuss the Fourier coefficients of the automorphic theta representa-tion Θ ( r )2 l . These coefficients, obtained by integrating functions in the representation spaceagainst certain characters of unipotent groups, are indexed by unipotent orbits, which maybe described by means of certain partitions of 2 l ; see, for example, [G] for more information.The set of unipotent orbits of Sp l is a partially ordered set. Let O (Θ ( r )2 l ) denote the set ofunipotent orbits O which are maximal with respect to the property that Θ ( r )2 l has a non-zeroFourier coefficient with respect to O . We expect that this set is a singleton. More precisely,we make the following conjecture. Conjecture 1.
Let r > be an odd integer, and write l = αr + β where ≤ β < r . Let O c (Θ ( r )2 l ) denote the unipotent orbit O c (Θ ( r )2 l ) = ( ( r α β ) if α is even ( r α − ( r − β + 1)) if α is odd.Then O (Θ ( r )2 l ) = {O c (Θ ( r )2 l ) } . This conjecture is also presented in [F-G2]. Gao and Tsai [G-T] have recently generalizedConjecture 1 to other groups, and also given an archimedean analogue. The compatibilityof their conjecture with Conjecture 1 is established in Section 4 of their paper.Although we are not able to prove Conjecture 1 in general, we can prove it in some cases,as follows.
Theorem 1. (1)
For all positive integers l , if O ∈ O (Θ ( r )2 l ) , then O ≤ O c (Θ ( r )2 l ) . (2) Assume that l = 0 , , , r − , r − , r − . Let n denote a non-negative integer, andassume that if l = 0 , then n ≥ . Then Conjecture 1 holds for the group Sp ( r )2( l + nr ) ( A ) .In particular, when r = 3 , the conjecture holds for all l and n . The first case for which we cannot prove Conjecture 1 is the group Sp (7)6 ( A ), where theconjecture states that Θ (7)6 is generic. The unramified constituents of Θ (7)6 are generic by[Gao2], but we do not know of a global result.We start with the vanishing property of the representations Θ ( r )2 l . The proof requires somelocal preparations. If U is any unipotent subgroup of a p -adic group or a metaplectic cover G , λ is a character of U , and ( π, V ) is a representation of G , then the Jacquet module J U,λ ( π ) is the quotient of V by the subspace spanned by vectors of the form π ( u ) v − λ ( u ) v with u ∈ U , v ∈ V . When λ is nontrivial, sometimes we refer to this as the twisted Jacquet odule. When λ is trivial we omit it from the notation, and denote this quotient as J U ( π ),the untwisted Jacquet module. The map V J U,λ ( π ) is the Jacquet functor.Let ν be a finite place of F , Θ ( r ) GL m ,ν denote the local theta representation on the r -foldcover of GL m ( F ν ) treated by Kazhdan and Patterson [K-P], Section I, with c = 0 (or thecorresponding character if m = 1), and Θ ( r )2 l,ν be the local theta representation for Sp ( r )2 l ( F ν ).Then we have the following proposition. Proposition 1.
Let P be the unipotent radical of Sp l whose Levi factor is GL m × Sp l − m , ≤ m ≤ l , and let U be its unipotent radical. Then J U (cid:16) Θ ( r )2 n,ν (cid:17) = Θ ( r ) GL m ,ν ⊗ Θ ( r )2 l − m,ν . Here the factor Θ ( r )2 l − m,ν is omitted if m = l . The proof of this result follows exactly the proof of [B-F-G1], Theorem 2.3, which givesthe same statement in a slightly different situation, and so is omitted here. There is also aglobal analogue; see for example [F-G2], Proposition 1.The proof of vanishing requires two lemmas that are cases of more general results. Thefirst, due to Gao [Gao2], gives information about when an unramified local constituent ofΘ ( r )2 l fails to be generic. (Similar information for covers of the general linear group is due toKazhdan and Patterson [K-P]; see Theorem I.3.5 there.) Lemma 1 (Gao) . Suppose that l > r . Then the local theta representation Θ ( r )2 l,ν at anunramfied place ν is not generic. This follows directly from [Gao2] by combining (the more general) Theorem 1.1 withProposition 5.1 there.The second is the following result, which is a special case of Leslie [L], Proposition 6.5.
Lemma 2 (Leslie) . Let π be a non-zero smooth admissible genuine representation of Sp (2 r )2 l ( F ν ) that is unramified. Then π is not supercuspidal. We now give our result on the vanishing of Fourier coefficients for Θ ( r )2 l . Proposition 2.
Suppose that O is a unipotent orbit which is greater than O c (Θ ( r )2 l ) or thatis not related to O c (Θ ( r )2 l ) . Then the representation Θ ( r )2 l has no non-zero Fourier coefficientcorresponding to O . Note that Proposition 2 is equivalent to the first part of Theorem 1.
Proof.
Recall that we write 2 l = αr + β with 0 ≤ β < r . If α = 0 then O c (Θ ( r )2 l ) = ( β ), so O ≤ O c (Θ ( r )2 l ) for all orbits O , and there is nothing to prove. So suppose that α >
0. Let O be a unipotent orbit satisfying the conditions of Proposition 2, let U O be the correspondingupper triangular unipotent subgroup of Sp l , and let ψ O denote any character of U O suchthat the Fourier coefficient(4.1) Z [ U O ] φ ( ug ) ψ O ( u ) du orresponds to the unipotent orbit O (see for example [B-F-G1], Section 4). (This is a slightabuse of notation as ψ O is not uniquely determined by O .) We need to prove that integral(4.1) is zero for all vectors φ in the space of Θ ( r )2 l and all g ∈ Sp l ( A ).First we show that it is enough to prove the vanishing of the coefficients for O k :=((2 k )1 l − k ) with 2 k > r . Indeed, by [G-R-S3], Lemma 2.6, this implies the vanishing for allorbits of the form O = ((2 k ) p e . . . p e d d ) with all p i ≤ k , e i ≥
0. This in term implies thevanishing for orbits of the form ((2 k + 1) p e . . . p e d d ) with all p i ≤ k + 1, e i ≥ ψ O -twisted Jacquet module at a good unramifiedplace is zero.We are reduced to showing the vanishing for O k = ((2 k )1 l − k ) with 2 k > r . It is enoughto prove the corresponding local statement. That is, we will show that at a good unramifiedfinite place ν (i.e. ord ν (2 r ) = 0), the ψ O k -twisted Jacquet module J V ( O k ) ,ψ O k ( π l ) is zero,where π l is the local constituent of Θ ( r )2 l at ν . (We use π l in place of Θ ( r )2 l,ν to simplify thenotation.) In particular π l is a genuine representation of the covering group Sp ( r )2 l ( F ν ) overthe nonarchimedean local field F ν . To establish the vanishing, we follow Leslie [L], Section8, who investigated a similar problem for the 4-fold cover. Let m be the integer such thatthe ψ O -twisted Jacquet module of π l for O = ((2 p )1 l − p ) is zero for p > m but such thatthe module for ((2 m )1 l − m ) is nonzero. Observe that m < l . Indeed, if m = l this wouldimply that π l is generic. Since 2 l > r , this contradicts Lemma 1.We also suppose by induction on l that the twisted Jacquet modules for π j , 1 ≤ j < l ,vanish on the orbits ((2 q )1 j − q ) when 2 q > r . Note that the base of the induction, that isthe case l = 2, is clear. We remark that even the case l = 2 requires 2 k > r . Indeed, if r = 3 then the theta representation for Sp (3)4 has a nonzero Fourier coefficient correspondingto (21 ), but not corresponding to (4) (see [F-G3], Lemma 2 and Proposition 3).The idea of the proof is to establish that the twisted Jacquet modules J V ( O k ) ,ψ O k ( π l )vanish by studying their descent properties. To prove that these modules vanish, it issufficient to show that the Fourier Jacobi modules F J m,α ( π l ) defined in [L], Section 6 (here α ∈ F × ν ), all vanish ([L], Corollary 6.4; as noted there this is the local version of [G-R-S3],Lemma 1.1). To do so, we will show that each Fourier Jacobi module F J m,α ( π l ) is asupercuspidal representation of Sp (2 r )2 l − m ( F ν ). Since l > m , if it were nonzero this wouldcontradict Lemma 2. Note that in taking the Fourier Jacobi module the cover is doubledsince r is odd and the Fourier Jacobi module uses a twist by the Weil representation whichlives on the double cover, so F J m,α ( π l ) is a genuine unramified representation of the 2 r -foldcover Sp (2 r )2 l − m ( F ν ). Aside from this the proof that the vanishing of all F J m,α ( π l ) implies thevanishing of J V ( O k ) ,ψ O k ( π l ) is the same as that in [L]. (Fourier Jacobi modules and descentsare also discussed in [G-R-S4], Section 3.8 and following.)To prove that the representation F J m,α ( π l ) is supercuspidal we must check that its un-twisted Jacquet functors are all zero. For 1 ≤ a ≤ l − m , let J a be the (untwisted) Jacquetfunctor on admissible representations of Sp (2 r )2 l − m ( F ν ) corresponding to the unipotent radicalof the parabolic subgroup of Sp l − m with Levi component GL a × Sp l − m − a . Then to provesupercuspidality, we must establish the vanishing of J a ( F J m,α ( π l )), 1 ≤ a ≤ l − m . Afterconjugating by a suitable Weyl element, this Jacquet module is isomorphic to the ψ a -twisted acquet module of π l with respect to the subgroup U a ( F ν ) := I a X Yv X ∗ I a where X ∈ Mat a × (2 l − a ) ( F ν ) has zero entries in the first column and v is upper triangularunipotent with center block I l − a ) , and where the character ψ a of U a ( F ν ) is given by ψ a ( u ) = ψ ( u a +1 ,a +2 + · · · + u a + m,a + m +1 + αu a + m, l − a − m +1 ) . Let H ⊂ Sp l be the subgroup of unipotent matrices which are zero above the main diagonalexcept on column a + 1 and row 2 l − a . Using the Geometrical Lemma of [B-Z], p. 448,if the module J U a ,ψ a ( π l ) is nonzero, then J H,ψ H ( J U a ,ψ a ( π l )) is nonzero for some character ψ H of H . There are two orbits under the action of GL a . Using Proposition 1, the modulewith ψ H = 1 factors through a coefficient of π l − a with respect to the orbit ((2 m )1 l − m − a ),and this vanishes by induction (note that since 2 l > r , also 2 m > r ). The non-trivialorbit contributes a similar Jacquet module with a replaced by a −
1. Under the action of GL a − there are again two orbits. As above the contribution from the trivial orbit is zeroby the induction hypothesis. Thus, we are again left with the non-trivial orbit. Repeating,we arrive at the nonvanishing of a twisted Jacquet module of π l with respect to the orbit((2 m + 2 a )1 l − m − a ). However, m was chosen to be maximal such that the module for((2 m )1 l − m ) is nonzero, and a ≥
1. We conclude that J U a ,ψ a ( π l ) = 0, and hence that allthe Jacquet modules J a ( F J m,α ( π l )) indeed vanish. Thus the representation F J m,α ( π l ) issupercuspidal, as claimed. This completes the proof of the vanishing. (cid:3) The local information developed above will also be used in treating the unramified corre-spondence in Section 6. We record the local vanishing in Proposition 7 below (there we useΘ ( r )2 l for the representation denoted Θ ( r )2 l,ν here).Our next proposition requires a result of Jiang and Liu ([J-L], Proposition 3.3) aboutthe Fourier coefficients of automorphic representations on symplectic groups and compositepartitions which we recall here for completeness. The notation is as in [J-L]. In fact the resultof Jiang and Liu is bit sharper since they give information about the characters supportingthe Fourier coefficients but we will not need this here. Their result is not stated for coveringgroups but the proof is identically the same. Proposition 3 (Jiang and Liu) . Let π be an irreducible automorphic representation of Sp n ( A ) or a metaplectic cover that is realized in the space of automorphic forms, and p = [(2 k + 1) p e p e · · · p e r r ] be a symplectic partition of n with k + 1 ≥ p ≥ p ≥ · · · ≥ p r ; e i = 1 if p i is even; and e i = 2 if p i is odd. Then π has a non-vanishing Fourier coeffi-cient attached to p if and only if it has a non-vanishing Fourier coefficient attached to thecomposite partition [(2 k + 1) n − k − ] ◦ [ p e p e · · · p e r r ] . Next we study the nonvanishing of the Fourier coefficients of theta functions. We startwith a proposition.
Proposition 4.
Suppose Conjecture 1 holds for a given r and l . Then it holds for r + l and l as well. roof. We know from Proposition 2 that if
O ∈ O (Θ ( r )2( l + r ) ) then O ≤ O c (Θ ( r )2( l + r ) ). Thus,we only need to prove that the representation Θ ( r )2( l + r ) has a non-zero Fourier coefficientcorresponding to the unipotent orbit O c (Θ ( r )2( l + r ) ).Write 2 l = αr + β with 0 ≤ β < r . Then 2( l + r ) = ( α + 2) r + β . By assumption, there isa unipotent subgroup U ( O c (Θ ( r )2 l )) of Sp l , and a character ψ U of the quotient [ U ( O c (Θ ( r )2 l ))]such that the Fourier coefficient(4.2) Z [ U ( O c (Θ ( r )2 l ))] ϕ ( r )2 l ( u ) ψ U ( u ) du corresponds to the unipotent orbit O c (Θ ( r )2 l ) and is not zero for some ϕ ( r )2 l in the space ofΘ ( r )2 l . It follows from [F-G2], Proposition 2.1, or as in Offen and Sayag [O-S], p. 10, thatthe constant terms of theta functions with respect to the unipotent radical of a Levi in factrealize products of theta functions on the Levi. (In [O-S], this is formulated in terms of thesurjectivity of an intertwining operator.) This is the global analogue of Proposition 1 above.We conclude that there is a function ϕ ( r )2( l + r ) in the space of Θ ( r )2( l + r ) such that the integral(4.3) Z [ U ( O c (Θ ( r )2 l ))] Z [ V ] Z [ V ] ϕ ( r )2( l + r ) v v u v ∗ ψ V ( v ) ψ U ( u ) dv dv du is not zero. Here V is the unipotent radical of the maximal parabolic subgroup of Sp l + r ) whose Levi part is GL r × Sp l , V is the maximal unipotent subgroup of GL r consisting ofupper unipotent matrices, and the character ψ V is the Whittaker character defined on thegroup V .To relate the Fourier coefficient in equation (4.3), to the Fourier coefficient correspondingto the unipotent orbit O c (Θ ( r )2( l + r ) ), we first use the result of Jiang and Liu recalled in Propo-sition 3. We deduce from this that the Fourier coefficient corresponding to the unipotentorbit O c (Θ ( r )2( l + r ) ) is not zero for some choice of data, if and only if the Fourier coefficientcorresponding to the orbit ( r l ) ◦ O c (Θ ( r )2 l ) is not zero for some choice of data. The Fouriercoefficient corresponding to the last orbit can be written as follows(4.4) Z [ U ,r ,l +1 ] Z [ U ( O c (Θ ( r )2 l ))] ϕ ( r )2( l + r ) v I r u I r ψ ′ U ,r ,l +1 ( v ) ψ U ( u ) du dv. We recall that the group U ,r ,l +1 was defined right before equation (2.1). Also, the character ψ U was defined in equation (4.2). The character ψ ′ U ,r ,l +1 is defined as follows. For v ∈ U ,r ,l +1 consider its factorization given by equation (2.3). Then we define ψ ′ U ,r ,l +1 ( v ) = ψ U ,r ,l +1 ( v ) ψ ′ V ( v ) where ψ U ,r ,l +1 is defined in equation (3.1) and ψ ′ V ( v ) is defined as follows.Let Y = ( y i,j ) ∈ Mat × l +1) . Define ψ ′ ( Y ) = ψ ( y , + y , l +1) ). Then, for v as in equation(2.3) we define ψ ′ V ( v ) = ψ ′ ( Y ).Using Proposition 2 we claim that integral (4.3) is not zero for some choice of data if andonly if integral (4.4) is not zero for some choice of data. Since integral (4.3) is not zero forsome choice of data, this claim will imply the Proposition. he proof of the claim is standard, and follows in a similar way as the proof of Proposition3.3 in [J-L]. We give some details. Let w denote the monomial matrix in the Weyl group of Sp n + l ) defined as w = ǫ ǫ I l ǫ ǫ ǫ i ∈ Mat r Here ǫ ( i, i −
1) = 1 for all 1 ≤ i ≤ ( r + 1) /
2, and ǫ ( i + ( r + 3) / , i + 2) = 1 for all0 ≤ i ≤ ( r − /
2. This determines w uniquely. We have ϕ ( r )2( l + r ) ( h ) = ϕ ( r )2( l + r ) ( wh ). Hence,after conjugating w to the right, integral (4.3) is not zero for some choice of data if and onlyif the integral Z [ U ( O c (Θ ( r )2 l ))] Z [ V ] ϕ ( r )2( l + r ) I r A BI l A ∗ I r v u v ∗ I r C I l D C ∗ I r ψ V ( v ) ψ U ( u ) du dv is not zero for some choice of data. Here, the matrices A , B , C and D are suitable matriceswhich are obtained in a similar way as in the proof of Proposition 3.3 in [J-L]. We omit thedetails. Performing a root exchange as in [G-R-S4], Section 7.1, and using Proposition 2, wededuce that the above integral is not zero for some choice of data if and only if integral (4.4)is not zero for some choice of data. (cid:3) It follows from Proposition 4 that given an odd number r , to prove Conjecture 1 for allsymplectic groups Sp ( r )2 l it is enough to prove the result for the symplectic groups Sp ( r )2 l with1 ≤ l ≤ r − ≤ l < r , Conjecture 1 states that O (Θ ( r )2 l ) = { (2 l ) } , that is, that thetheta representation is generic. When r < l < r −
1, the conjecture is that O (Θ ( r )2 l ) = { (( r − l − r + 1)) } . The next Proposition gives a lower bound for some element of theset O (Θ ( r )2 l ) (or for the associated orbit if this set is, as expected, a singleton). Proposition 5.
Assume that ≤ l ≤ r . Then the representation Θ ( r )2 l has a non-zero Fouriercoefficient corresponding to the unipotent orbit ( l ) .Proof. Let V l denote the unipotent radical of the parabolic subgroup of Sp l whose Levi partis isomorphic to GL l/ if l is even and to GL ( l +1) / if l is odd. Let ψ V l denote the characterof [ V l ] given by ψ V l ( v ) = ψ ( P l − i =1 v i,i +2 ). To prove the proposition we will assume that theintegral(4.5) Z [ V l ] ϕ ( r ) ( vg ) ψ V l ( v ) dv is zero for all functions ϕ ( r ) in Θ ( r )2 l and g ∈ Sp ( r )2 l ( A ) and derive a contradiction. We maytake g = e . Let w ∈ Sp l ( F ) be the monomial matrix with non-zero entries ± w i, i − = 1 for all 1 ≤ i ≤ l . Since w ∈ Sp l ( F ), we have ϕ ( r ) ( v ) = ϕ ( r ) ( wv ). Conjugating w across v , we deduce that the integral(4.6) Z ϕ ( r ) (cid:18)(cid:18) I l AI l (cid:19) (cid:18) B B ∗ (cid:19) (cid:18) I l C I l (cid:19)(cid:19) ψ ( B ) dA dB dC s zero for all choices of data. Here A and C are integrated over [Mat l, ] where Mat l, is thesubgroup of Mat l consisting of all matrices X = ( X i,j ) ∈ Mat l such that X i,j = 0 for all i ≥ j (the group Mat l was defined at the beginning of Section 2). The variable B is integrated over[ L l ] where L l is the unipotent subgroup of GL l consisting of all upper triangular matrices,and ψ is the Whittaker character defined on L l .Integral (4.6) is a special case of the situation dealt with in [J-L] Propositions 3.2 and 3.3.Performing root exchange as in [J-L] (see also [G-R-S4], Section 7.1), and using the fact thatif an automorphic function is zero then all its Fourier coefficients are zero, we deduce thatthe integral Z ϕ ( r ) (cid:18)(cid:18) I l AI l (cid:19) (cid:18) B B ∗ (cid:19)(cid:19) ψ ( B ) dA dB is zero for all choices of data. Here A is integrated over the quotient [Mat l ], and B isintegrated as in (4.6). Applying [F-G2], Proposition 2.1 (or, again, as in [O-S] p. 10), wededuce that the Whittaker coefficient of the theta representation of the group GL ( r ) l ( A ) iszero for all choices of data. However, since 1 ≤ l ≤ r it follows from [K-P] that this thetarepresentation on GL ( r ) l ( A ) is generic. Thus we have derived a contradiction. (cid:3) Denote by e i,j the square matrix whose ( i, j ) entry is one and with zeros elsewhere. Given l ≥ ≤ i, j ≤ l , let e ′ i,j = e i,j ± e l − j +1 , l − i +1 , with the sign determined so that thematrix I l + e ′ i,j is in Sp l . Also, let E i, l +1 − i = { I l + te i, l +1 − i } , E ′ i,j = { I l + te ′ i,j } ( i + j = 2 l + 1)be the associated one parameter unipotent subgroups, corresponding to the long (resp. short)roots of Sp l . Proposition 6.
Suppose that ≤ l ≤ r − . (1) If l is odd, then the set O (Θ ( r )2 l ) contains an orbit which is greater than or equal tothe unipotent orbit (( l + 1)( l − . (2) Suppose l is even, l = r − and that O ≥ ( l ) for all O ∈ O (Θ ( r )2 l ) . Then O (Θ ( r )2 l ) isa singleton and O (Θ ( r )2 l ) ≥ (( l + 2)( l − .Proof. Consider first the case that l is odd. By Proposition 5, the Fourier coefficient (4.5) isnonzero. However, the stabilizer of the unipotent orbit ( l ) in Sp l is SL , which embeds inthe parabolic subgroup with Levi isomorphic to GL ( l +1) / by the diagonal embedding. Notethat since 1 ≤ l ≤ r − l is odd, in fact 1 ≤ l ≤ r −
2. Thus the cover Sp ( r )2 l ( A ) restrictsvia this embedding to a group isomorphic to SL ( r )2 ( A ). The integral (4.5) then gives anautomorphic function of g ∈ SL ( r )2 ( A ) which is genuine. Therefore, it can not be constant,and so this function has a nontrivial Whittaker coefficient.For y ∈ A , let x ( y ) ∈ Sp l ( A ) be given by x ( y ) = I l + P ( l − / i =1 ye ′ i − , i + ye l,l +1 . Weconclude that there a choice of data such that the integral(4.7) Z [ V l ] Z [ G a ] ϕ ( r ) ( vx ( y ) g ) ψ V l ( v ) ψ ( αy ) dy dv is not zero. Here α ∈ F ∗ . Then is not hard to check that, after root exchange, the integral(4.7) has as inner integration a Fourier coefficient corresponding to the unipotent orbit ( l + 1)( l − l is even. The argument is different since we do nothave unipotent elements inside the stabilizer of this unipotent orbit. However, the stabi-lizer contains the subgroup generated by the matrices t ( a ) = diag( a, a − , . . . , a, a − ) and w = diag( J , . . . , J ), with J defined in Section 2. Recall that the elements e t ( a ) =( t ( a ) , η − ( t ( a ))) ∈ Sp ( r )2 l ( A ) were introduced at the end of Section 2.Let a ∈ A ∗ be an r -th power, and define(4.8) Λ( a ) = Z [ V l ] ϕ ( r ) ( v e t ( a ) w ) ψ V l ( v ) dv. Denote also by Λ ( e ) integral (4.5) with g = e . Since w stabilizes the character ψ V l we haveΛ( e ) = Λ ( e ).Performing Fourier expansions and root exchanges similar to those in the proof of Propo-sition 5, we obtain that Λ( a ) is equal to(4.9) Z ϕ ( r ) (cid:18)(cid:18) I l AI l (cid:19) (cid:18) B B ∗ (cid:19) (cid:18) I l C I l (cid:19) e h ( a ) ww (cid:19) ψ ( B ) dA dB dC. Here, A is integrated over [Mat l ], B is integrated as in (4.6), and C is integrated overMat l, ( A ) (defined after (4.6)). The Weyl element w was defined following (4.5), and e h ( a ) = w e t ( a ) w − = ( h ( a ) , η ( t ( a )) − ), where h ( a ) = diag( aI l , a − I l ). We want to emphasize thedifference between the computation performed here, and the one performed in Proposition5. In that Proposition we needed to show that a certain integral vanished for all choices ofdata. In this Proposition we need a precise identity. This is why we made the assumptionthat O ≥ ( l ) for all O ∈ O (Θ ( r )2 l ). Indeed, this assumption implies that the set O (Θ ( r )2 l ) is asingleton, and that the representation Θ ( r )2 l has no non-zero Fourier coefficient correspondingto unipotent orbits which are not related to ( l ), for example the orbit (( l + 2)1 l − ). To provethat Λ( a ) is equal to integral (4.9), we need to make use of this.Recall that a is an r -th power. In (4.9) conjugate the matrix e h ( a ) to the left. First, weget the factor | h | − l / from the change in variables in C . The torus h ( a ) commutes with thematrix diag( B, B ∗ ). We are left with the computation of the constant term consisting of allmatrices ( I AI ) where A ∈ Mat l . It follows from [F-G2] Proposition 2.1 that we obtain thefactor of χ Sp ( r )2 l , Θ ( e h ( a )). By the formula in [F-G2], top of p. 93, this last term is equal to | a | l ( l +1)( r − / r . Putting this together, we have proved that(4.10) Λ( a ) = | a | l ( l +1)( r − r − l Z ϕ ( r ) (cid:18)(cid:18) I l AI l (cid:19) (cid:18) B B ∗ (cid:19) (cid:18) I l C I l (cid:19) ww (cid:19) ψ ( B ) dA dB dC. Hence, Λ( a ) = | a | l ( l +1)( r − r − l Λ( e ) = | a | l ( l +1)( r − r − l Λ ( e ).Next we compute Λ( a ) in a different way. Going back to the definition in (4.8), we firstconjugate w to the left. We obtainΛ( a ) = Z [ V l ] ϕ ( r ) ( v e t ( a − )) ψ V l ( v ) dv. epeating the same computations we performed in equations (4.9) and (4.10), we obtainΛ( a ) = | a | − l ( l +1)( r − r + l Λ ( e ). By Proposition 5, there is a choice of data such that Λ ( e ) isnot zero. Hence, we must have l ( l +1)( r − r = l . This is equivalent to l = r −
1, so the secondpart follows. (cid:3)
The first part of Theorem 1 is in Proposition 2. We now give the proof of second part ofthe Theorem.
Proof.
Consider first the case l = 0. Then applying Proposition 4, it is enough to prove thetheorem for the group Sp ( r )2 r ( A ). According to Conjecture 1, we need to prove that Θ ( r )2 r hasa non-zero Fourier coefficient corresponding to the unipotent orbit ( r ). This was proved inProposition 5.For all other cases stated in Theorem 1, part 2, using Proposition 4, it is enough to provethe result for the group Sp ( r )2 l ( A ). The case l = 1 is clear. Next consider l = 2. If r = 3, itfollows from Proposition 2 that Θ (3)4 is not generic. From Proposition 4, Θ (3)4 has a nonzeroFourier coefficient corresponding to the orbit (2 ). If r ≥
5, it follows from Proposition 6that O (Θ ( r )4 ) is greater than (2 ). Hence Θ ( r )4 is generic and we are done.When l = r −
2, the Conjecture states that O (Θ ( r )2( r − ) = { (( r − r − } . This followsfrom Proposition 6. When l = r −
1, the Conjecture states that O (Θ ( r )2( r − ) = { (( r − ) } .This follows from Proposition 5. Thus the second part of the Theorem is proved. (cid:3) Cuspidality of the Lift
In this Section we discuss the cuspidality of the representation σ ( r ) n,k . The main result isthat the first non-zero occurrence of the generalized theta lift is automatically cuspidal. Thisgeneralizes Rallis’s tower property, and is found in Theorem 2 below. The proof requiresshowing the vanishing of constant terms. We will establish this by considering various Fourierexpansions, using the process of root exchange (see [G-R-S4, Section 7.1]), and makinguse of two key ingredients: the smallness of the representation Θ ( r )2 n + k ( r − established inProposition 2, and the cuspidality of the representation π ( κr ) .We begin with several Lemmas. Let U m be the unipotent subgroup U m = U ,m +1 ,n + kr − m − ,and let ψ U m denote the character ψ U m = ψ U ,m +1 ,n + kr − m − of this group (see (3.1)). The firstLemma is closely related to Lemma 2.2 in [G-R-S3]. Lemma 3.
Suppose that r ≤ m ≤ n + kr − . Then the integral (5.1) Z [ U m ] θ ( r )2 n + k ( r − ( u ) ψ U m ( u ) du is zero for all choices of data.Proof. Let x ( p ) = I + pe m +1 , n + k ( r − − m , with I the identity matrix of size 2 n + k ( r − { x ( p ) } . This is a sum of integrals against characters ψ ( αp ), α ∈ F . The nontrivial terms contribute zero. Indeed, the Fourier coefficient we obtainfrom a nontrivial term corresponds to the unipotent orbit ((2 m + 2)1 k ( r − − m − ). Since m ≥ r , this last unipotent orbit is not comparable with the unipotent orbit O c (Θ ( r )2 n + k ( r − )defined in Conjecture 1 above. By Proposition 2, these Fourier coefficients vanish. e are left with the contribution from α = 0. That is, integral (5.1) is equal to(5.2) Z [ U m ] Z [ G a ] θ ( r )2 n + k ( r − ( ux ( p )) ψ U m ( u ) dp du. The quotient group { x ( p ) } U m \ U m +1 can be identified with a row vector of size 2( n − m −
1) + k ( r − Sp n − m − k ( r − ( F ). The trivial orbit contributes zero to integral (5.2).Indeed, to prove this, we use Proposition 1 in [F-G2], which identifies the constant term ofa theta function with lower rank theta functions, with a there equal to m + 1. This impliesthat as an inner integration we obtain the Whittaker coefficient of the theta function definedon an r -fold cover of GL m +1 ( A ). Since m + 1 > r , it follows from [K-P] that this Whittakercoefficient is zero. We conclude that the integral (5.1) is a sum of integrals of the form Z [ U m +1 ] θ ( r )2 n + k ( r − ( u ) ψ U m +1 ( u ) du. Applying induction, these integrals are all zero. Hence integral (5.1) is zero for all choices ofdata. (cid:3)
In the next Lemma we establish another vanishing result. Assume again that r ≤ m ≤ n + kr −
1. Let V m denote the maximal unipotent subgroup of GL m +1 consisting of uppertriangular unipotent matrices. Let Mat m +1 denote the subgroup of Mat m +1 of all matrices x ∈ Mat m +1 such that x i,j = 0 for all i ≥ j . Let U m denote the subgroup of U m whichconsists of all matrices t ( v, x ) = v xI n + kr − m − v ∗ v ∈ V m , x ∈ Mat m +1 . By restriction, the character ψ U m is a character of U m . Lemma 4.
Assume that r ≤ m ≤ n + kr − . Then the integral (5.3) Z [ U m ] θ ( r )2 n + k ( r − ( u ) ψ U m ( u ) du is zero for all choices of data.Proof. We start by defining some unipotent subgroups of Sp n + k ( r − . Let a = 2( n + kr − m − b ( m ) = c ( m ) = m/ m is even, and b ( m ) = ( m + 1) / c ( m ) = ( m − / m is odd. For 1 ≤ j ≤ b ( m ) let Y j be the upper triangular unipotent subgroup Y j = { y j ( p , . . . , p a + j ) = I n + kr ) + a + j X i =1 p i e ′ j,m + i +1 } , and for 1 ≤ j ≤ c ( m ) let Y ′ j be the upper triangular unipotent subgroup Y ′ j = { y ′ j ( p , . . . , p a + c ( m ) − j +1 , q ) = I n + kr ) + a + c ( m ) − j +1 X i =1 p i e ′ b ( m )+ j,m + i +1 + qe b ( m )+ j,a + c ( m )+ m − j +3 } . e also define corresponding lower unipotent groups. For 1 ≤ j ≤ c ( m ) let Z j = { z j ( p , . . . , p a + j ) = I n + kr ) + a + j + m +1 X i = m +2 p i e ′ j,j +1 } , and for 1 ≤ j ≤ b ( m ) let Z ′ j = { z ′ j ( p , . . . , p a + b ( m ) − j +1 ) == I n + kr ) + a + b ( m ) − j X i =1 p i e ′ i,c ( m )+ j +1 + p a + b ( m ) − j +1 e a + b ( m ) − j +1 ,c ( m )+ j +1 } . To prove the Lemma, we expand (5.3) along the quotient [ Y ]. Doing so, we see that theintegral (5.3) is equal to(5.4) X ξ i ∈ F Z [ Y ] Z [ U m ] θ ( r )2 n + k ( r − ( y ( p , . . . , p a +1 ) u ) ψ U m ( u ) ψ ( X ξ i p i ) dy du. Since the function θ ( r )2 n + k ( r − is automorphic, for ξ j ∈ F we have θ ( r )2 n + k ( r − ( z ( − ξ , . . . , − ξ a +1 ) h ) = θ ( r )2 n + k ( r − ( h ) . Using this in the integral (5.4) and then conjugating the matrix z ( − ξ , . . . , − ξ a +1 ) to theright, we obtain (after a change of variables in u ),(5.5) X ξ i ∈ F Z [ Y ] Z [ U m ] θ ( r )2 n + k ( r − ( y ( p , . . . , p a +1 ) uz ( − ξ , . . . , − ξ a +1 )) ψ U m ( u ) dy du. Thus if we prove that the inner integration in (5.5) is zero for all choices of data, this willimply that the integral (5.3) is zero for all choices of data.Now we repeat this process with the inner integration of (5.5), this time using the groups Y and Z . The process is visibly inductive, and we repeat it b ( m ) times. More precisely,for all 1 ≤ j ≤ b ( m ), we expand the corresponding integral along the group Y j , and use thegroup Z j as above unless j = b ( m ), m odd, in which case we use the group Z ′ . Let U m denote the unipotent group generated by U m and by all Y j for 1 ≤ j ≤ b ( m ). Then thisshows that if the integral(5.6) Z [ U m ] θ ( r )2 n + k ( r − ( u ) ψ U m ( u ) du is zero for all choices of data, then the integral (5.3) is zero for all choices of data. Here ψ U m is a character of U m obtained from U m by extending it trivially.Next, expand the integral (5.6) along the unipotent abelian group { x ′ ( p ) = I n + kr ) + pe b ( m )+1 ,a + c ( m )+ m +2 } . We claim that the contribution to the expansion from the non-constant terms is zero. Indeed,for these terms we obtain a Fourier coefficient which corresponds to the unipotent orbit((2 b ( m ) + 2)1 n + kr − b ( m ) − ). Since r ≤ m , it follows that this unipotent orbit is not related o the orbit O c (Θ ( r )2 n + k ( r − ). Hence by Proposition 2 these coefficients vanish. Thus theLemma will follow once we prove that the integral(5.7) Z [ G a ] Z [ U m ] θ ( r )2 n + k ( r − ( x ′ ( p ) u ) ψ U m ( u ) dp du is zero for all choices of data. To show this, observe that the group { x ′ ( p ) } is the center of thegroup Y ′ . Thus we can expand the integral (5.7) along the quotient Y ′ / { x ′ ( p ) } . Dependingon the parity of m , we use Z ′ or Z ′ as above. Once again the argument is inductive, andafter we carry it out with the groups Y ′ j for 1 ≤ j ≤ c ( m ), we obtain the integral (5.1). ByLemma 3, since r ≤ m , this integral is zero for all choices of data. Lemma 4 follows. (cid:3) For the next Lemma, let α and l be two positive integers which satisfy 2 ≤ α ≤ k , and( r − / < l ≤ r −
1. We work with the unipotent group U a,b,c with a = α, b = l and c = n + kr − lα . Consider the Fourier coefficient(5.8) f ( g ) = Z [ U α,l,n + kr − lα ] θ ( r )2 n + k ( r − ( ug ) ψ U α,l,n + kr − lα ( u ) du, g ∈ Sp ( r )2 n + k ( r − ( A ) . For short we shall write ψ for ψ U α,l,n + kr − lα . Because of the factorization in equation (2.3),we may view (5.8) as a function of u ′ α,n + kr − lα (0 , Z ) where Z ∈ Mat α ( A ). We have Lemma 5.
For α and l in the range specified above, the Fourier coefficient (5.8) is invariantunder the adelic points of the group { u ′ α,n + kr − lα (0 , Z ) } . That is, for all Z ∈ Mat α ( A ) wehave f ( u ′ α,n + kr − lα (0 , Z ) g ) = f ( g ) .Proof. Since the group of all matrices u ′ α,n + kr − lα (0 , Z ) is an abelian subgroup of Sp n + k ( r − ,we can expand the function f ( g ) along it. We obtain(5.9) f ( g ) = X γ ∈ Mat α ( F ) Z [Mat α ] f ( u ′ α,n + kr − lα (0 , Z ) g ) ψ γ ( Z ) dZ, where γ → ψ γ is an isomorphism of the abelian group Mat α ( F ) with its dual. We need toprove that the nontrivial characters contribute zero to the expansion. The group GL α ( F ),embedded in Sp n + k ( r − ( F ) by the map δ diag( δ, δ, . . . , δ, I, δ ∗ , . . . , δ ∗ ), acts on the set { γ } . Here δ ∈ GL α ( F ) appears l times. It is enough to consider representatives under thisaction. If γ is not zero then there are two cases to consider. The first case is when ψ γ ( x ( p )) isnot zero for x ( p ) = I n + k ( r − + pe j ,j where j = ( l − α +1 and j = 2 n + k ( r − − ( l − α − ψ γ ( x ( p )) is not zero for x ( p ) = I n + k ( r − + pe ′ j ,j where j =( l − α + 1 and j = 2 n + k ( r − − lα + 1.We start with the first case. Let w be the Weyl group element w = diag( w, I n + kr − lα ) , w ∗ )in Sp n + k ( r − ( F ). Here w in GL lα ( F ) is the matrix whose only nonzero entries are 1 in po-sitions ( j, ( j − α + 1) for 1 ≤ j ≤ l and positions ( j , j ) for j = l + ( j − α + a − j + 1and j = ( j − α + a + 1 with 1 ≤ a ≤ α − ≤ j ≤ l .Conjugating the integral in equation (5.9) by w , we obtain the integral(5.10) Z [ X ] Z [ U ′ l ] Z [ G a ] θ ( r )2 n + k ( r − ( uu ′ ,n + kr − l (0 , m ) x ) ψ U l ( u ) ψ ( βm ) dm du dx s an inner integration to the integral (5.9). Here β ∈ F ∗ , and U ′ l is the subgroup of U ,l +1 ,n + kr − l − defined as follows. An element u = ( u i,j ) ∈ U ′ l if u i,j = 0 for all 1 ≤ i ≤ l − l + 1 ≤ j ≤ l + i ( α − X , it consists of all matrices of the form(5.11) x = I l y I l ( α − I I l ( α − y ∗ I l where y ∈ Mat l ( α − × l satisfies the conditions y i,j = 0 for all ( i, j ) such that 1 ≤ j ≤ l and( j − α −
1) + 1 ≤ i ≤ l ( α − Z [ U l ] Z [ G a ] θ ( r )2 n + k ( r − ( uu ′ ,n + kr − l (0 , m )) ψ U l ( u ) ψ ( βm ) dm du dx for all choices of data. To prove this claim, for 1 ≤ j ≤ l − V j denote the unipotent subgroup of U l defined by V j = { x i ( p i ) = I n + k ( r − + p i e ′ j,l + i , : 1 ≤ i ≤ j ( α − } , and let X j denote the unipotent subgroup of X defined by matrices of the form (5.11) withall entries of y equal to zero outside the ( j + 1)-st column. We proceed inductively, startingwith j = 1. We expand the integral (5.5) along the group [ V j ], and then we perform rootexchange with the group X j . After the root exchange corresponding to j = l −
1, we obtainthe integral (5.12) as an inner integration to integral (5.10).However, the integral (5.12) corresponds to the unipotent orbit ((2 l +2)1 n + kr − l − ). Since( r − / < l , this unipotent orbit is not comparable with the unipotent orbit O c (Θ ( r )2( n + kr ) )in Conjecture 1. Hence Proposition 2 implies that (5.12) is zero for all choices of data. Thiscompletes the first case of representative in (5.9).Next we consider the second case. For this case we use a different Weyl group element in Sp n + kr ) ( F ), which we denote by w . To define w , we set w i, ( i − α +1 = w l + i, n + k ( r − − ( l − i +1) α =1 for 1 ≤ i ≤ l . Then we extend it in an arbitrary way to a Weyl group element of Sp n + kr ) ( F ). Conjugating the corresponding integral in the expansion (5.9), we obtainintegral (5.3) with m = 2 l − (cid:3) With this preparation, we now establish the tower property for this theta lift. Fix k ≥ Theorem 2.
Suppose that the representation σ ( r ) n,m is zero (i.e. every function in this space isidentically zero) for all m , ≤ m < k , m ≡ k mod 2 . Then σ ( r ) n,k is a cuspidal representation.Proof. To prove the cuspidality of the lift, we need to show that the constant terms of therepresentation σ ( r ) n,k along any unipotent radical of a standard maximal parabolic subgroup of O k are zero for all choices of data. These are the subgroups N α , 1 ≤ α ≤ [ k/ SO k of the form I α ∗ ∗ I β ∗ I α , β = k − α. Thus, with f ( h ) given by (3.3), we need to prove that the integral(5.13) Z [ N α ] f ( n α h ) dn α is zero for all choices of data.We start by unfolding the theta series θ (2) ,ψ nk , which is expressed as a sum over ξ ∈ F nk .We choose the polarization ξ = ( ξ , ξ ), where ξ ∈ F nα and ξ ∈ F nβ . Write ξ =( ξ , , ξ , , . . . , ξ ,α ) where ξ ,i ∈ F n . The action of Sp n is given by multiplication on theright on each ξ ,i . Now the projection l ( u ) that appears in (3.3) depends only on the valuesof l ( u ′ k,n ( Y, Z )) (for the notation, see (2.3)). Write(5.14) Y = y α y β y ′ α , Z = z α ∗ ∗ z z β ∗ z ∗ ∗ ∈ Mat k . Here y α , y ′ α ∈ Mat α × n and y β ∈ Mat β × n . Also, z α ∈ Mat α , z β ∈ Mat β , z ∈ Mat β × α and z ∈ Mat α . Using the formulas for the Weil representation ω ψ (see for example [G-R-S1,Section 1, part 3]), we obtain(5.15) θ (2) ,ψ nk ( l ( u ′ k,n ( Y, Z )) ι ( n α , g )) = X ξ ,ξ ω ψ (( ξ , l ( u ′ k,n ( Y, Z )) ι ( n α , g )) φ (0 , ξ )Here φ is a Schwartz function of A nk .From the definition of the homomorphism l , we have l ( u ′ k,n ( Y ξ , ξ ,
0) where Y ξ = (cid:0) ξ (cid:1) . In the integral (5.13) the variables y ′ α , defined in (5.14), are integrated over the quotientMat α × n ( F ) \ Mat α × n ( A ). Hence, after conjugating the element u ′ k,n ( Y ξ ,
0) to the right wemay combine summation with integration. It follows that to prove the vanishing of theintegral (5.13) for all choices of data, it suffices to prove the vanishing of the integral(5.16) Z [ Sp n ] Z ϕ ( κr ) ( i ( g )) θ (2) ,ψ n ( k − α ) ( l ( u ′ k,n ( Y, Z )) ι (1 , g )) θ ( r )2 n + k ( r − ( u ′ k,n ( Y, Z ) u ι ( n α , g )) ψ U k,r ,n ( u ) ψ α ( Z ) dY dZ du dn α dg for all choices of data, where the notation is as follows.In the coordinates of (5.14), l is defined by l ( u ′ k,n ( Y, Z )) = ( y β , tr( z β T k − α )) ∈ H nβ +1 (where the rows of y β are listed with the bottom row first, similarly to the definition of themap l in Section 2). The variable Y is integrated over [ Y ], where Y is the subgroup ofMat k × n given by all matrices Y as in (5.14) with y ′ α = 0, and Z is integrated over [Mat k × k ].Also, u is integrated over [ U k,r ,n ], and n α is integrated over [ N α ]. Using the coordinates ofequation (5.14), the character ψ α is defined as ψ α ( Z ) = ψ (tr( z α )). Also, notice that (when β >
0) the theta series appearing in (5.16) is defined on the double cover of Sp n ( k − α ) ( A ). his follows since after the above collapsing of summation and integration we are left withthe summation over ξ ∈ F n ( k − α ) . The rightmost argument of the theta series is ι (1 , g )since ω ψ ( ι ( n α , g )) φ (0 , ξ ) = ω ψ (( ι (1 , g )) φ (0 , ξ ). If k = β = 0 and k = 2 α then the thetafunction θ (2) ,ψ n ( k − α ) in (5.16) is omitted.The next step is to define a certain Weyl group element w ∈ Sp n + k ( r − ( F ), which we willthen use to conjugate the various groups. This Weyl element is of the form(5.17) w = w w I n w w where w i ∈ Mat kr . To specify it, it is enough to specify the matrices w and w . Thesematrices will be block matrices whose only nonzero entries are the identity matrices I α and I β . To describe the location of each such identity block, it is enough to specify the locationin each w j , j = 1 ,
2, of its first 1 on the diagonal. The matrix w has an identity matrix I α whose first 1 is at position ( α ( i −
1) + 1 , k ( i −
1) + 1) for 1 ≤ i ≤ r . Since w hasonly one non-zero entry in each row, note that this implies that the first αr rows of w are all zeros. Next, the matrix w has an identity matrix I α whose first 1 is at position( αr + α ( i −
1) + 1 , k ( i −
1) + 1) for 1 ≤ i ≤ r . This then implies that the correspondingrows in the matrix w are all zero. Finally, in w there is a block of I β whose first 1 is atposition ( α ( r −
1) + β ( i −
1) + 1 , k ( i −
1) + α + 1) for 1 ≤ i ≤ r .For example, when r = 7, we have w = I α β α α β α α β α α I α α I α α α α β I β β I β β I β , w = α β α α β α α β α α α I α α I α α I α β β β where all the blank entries are zero.Before conjugating by w , we perform a certain root exchange. To do that, let L α,β denotethe unipotent subgroup of GL k consisting of all matrices of the form(5.18) l = I α a bI β cI α and let L denote any unipotent subgroup of L α,β such L α,β = L N α . For example, one maychoose the group of all matrices l as above such that c = 0 and such that bJ α is strictlyupper triangular. Consider the direct sum L α,β ⊕ . . . ⊕ L α,β ⊕ L where L α,β appears r − Sp n + k ( r − as(5.19) diag( l , l , . . . , l r − , l , I n , l ∗ , l ∗ r − , . . . , l ∗ ) . e will also need to consider the subgroup of Mat k which consists of all matrices of the form l − = α a β b c α We denote this group by L − .Now we carry out root exchange with the embedded copies of the groups L α,β and L − . Webegin with L α,β is embedded in the first component of L α,β ⊕ . . . ⊕ L α,β ⊕ L and then inside Sp n + k ( r − as in (5.19). Since L α,β is not abelian, this root exchange needs to be carriedout in stages, as follows. First, we exchange the unipotent elements which are in the centerof L α,β , i.e. the (abelian) group of all matrices l in (5.18) such that a = c = 0, with thegroup of all matrices u k,r ,n ( l − ) (see (2.1)) such that l − ∈ L − with a = c = 0. After doingthis, we proceed with the abelian group consisting of all matrices l ∈ L α,β with b = c = 0,and then with the abelian group of all matrices with a = b = 0. Next, we exchange L α,β embedded in the second component of L α,β ⊕ . . . ⊕ L α,β ⊕ L and then inside Sp n + k ( r − asin (5.19). For this group we use the copy of L − embedded in Sp n + k ( r − as l − u k,r ,n ( l − ).We continue this process for all i with 1 ≤ i ≤ r −
1, exchanging the i -th copy of L α,β inside L α,β ⊕ . . . ⊕ L α,β ⊕ L with a subgroup of u ik,r ,n ( L − ). Then, we perform root exchangecorresponding to l − ∈ L − , embedded in U k,r ,n as all matrices l − u k,r ,n ( l − ) We exchangethis group with the group of all matrices u ′ k,n (0 , Z ) where(5.20) Z = z z z ∗ . After performing these root exchanges, we conjugate by the Weyl element w defined in(5.17). Thus, to prove that integral (5.16) is zero for all choice of data, we conclude that itis enough to prove that the integral(5.21) Z ϕ ( κr ) ( i ( g )) θ (2) ,ψ n ( k − α ) ( l ( u ) ι (1 , g )) ψ U β,r ,n ( u ) ψ V α,r − ( v ) θ ( r )2 n + k ( r − I α ( r − C DI C ∗ I α ( r − v u v ∗ I α ( r − A IB A ∗ I α ( r − ι (1 , g ) d (...)is zero for all choices of data. Here I denotes the identity matrix of size 2 n + ( k − α )( r − g is integrated as in the integral (5.16). The variable u is integrated over thequotient [ U β,r ,n ], where U β,r ,n ⊆ Sp n + β ( r − is embedded inside Sp n + k ( r − by the map u → diag( I α ( r − , u, I α ( r − ). Also, V α,r − denotes the subgroup of U α,r − ,n + βr generated byall matrices of the form Q r − i =1 u iα,r − ,n + βr ( X i ) with X i ∈ Mat α ; V α,r − is isomorphic to aunipotent subgroup of GL α ( r − . The character ψ V α,r − is the restriction of ψ U α,r − ,n + βr to V α,r − . The variable v is integrated over [ V α,r − ].Next we define the regions over which the variables A, B, C and D (each matrices of acertain size) are integrated. These are given by considering them as block matrices andimposing various conditions. We start with the variable D . Consider the subgroup D ⊂ at α ( r − consisting of block matrices with blocks of size α such that each ( i, j ) block with j < i is the zero matrix 0 α . Thus, for example if r − D consists of all matricesof the form D = X X X X α X X X ∗ α α X ∗ X ∗ α α α X ∗ X , X , X , X ∈ Mat α ; X , X ∈ Mat α . With these notations D is integrated over [ D ]. Similarly, let B ⊂ Mat α ( r − consist of blockmatrices with blocks of size α such that each ( i, j ) block with j ≤ i + 1 is the zero matrix0 α . For example if r − B consists of all matrices of the form B = α α X X X α α α X X ∗ α α α α X ∗ α α α α α α α α α α X , X ∈ Mat α ; X , X ∈ Mat α . Then B is integrated over [ B ].As for C , define a subgroup C ⊂ Mat α ( r − × (2 n +2 βr ) as follows. Write C = (cid:18) C C C C (cid:19) , with C ∈ Mat αr × n and C , C , C ∈ Mat αr × βr . Then C is the subgroup of such matricessuch that C and C may be written as block matrices, with blocks of size α × β , such thateach ( i, j ) block with j < i is 0 α × β . For example, when r = 3, then both C and C arematrices of the form X X X X X X X i ∈ Mat α × β , where the zeroes indicate the zero matrices of the corresponding sizes. The variable C isintegrated over [ C ].Finally, let A denote the subgroup of Mat (2 n + β ( r − × α ( r − consisting of matrices of theform A = A A A A (cid:18) A A (cid:19) ∈ Mat (2 n + βr ) × α ( r − , where the first column has width 2 α , the last row has height β , and A , A ∈ Mat β ( r − × α ( r − satisfy the following conditions. First viewing A as a block matrix with blocks of size α × β ,all ( i, j ) blocks with j < i are zero matrices. Second, viewing A similarly, all ( i, j ) blockswith j ≤ i are the zero matrix. For example, for r − A = X X X X X X X X X X , A = Y Y Y Y Y Y X i , Y j ∈ Mat β × α . hen A is integrated over [ A ].Our goal is to prove that the integral (5.21) is zero for all choices of data. To do that westart with a sequence of root exchanges, and a repeated application of Lemma 5. Recall thatin terms of blocks of height α , the matrices C and D each have r − , α × β for C and 0 α for D , and so on for the remaining rows.We first perform the root exchange using (1 ,
3) block in B and then using the (1 ,
3) blockof A . Note that there is a compatibility in the block sizes with the (2 ,
1) blocks of thematrices D and C , resp. Indeed, the (1 ,
3) block of B is of size α × α which is exactly thesize of the block in the (2 ,
1) position of D . Similarly, the (1 ,
3) block of A is of size β × α which is exactly what is needed for root exchange with the block matrix at the (2 ,
1) positionof the C , which is of size α × β . We continue this process with the third row of the blockmatrices C and D . For these two matrices the (3 ,
1) and (3 ,
2) blocks are each zero. Weintegrate over the (1 ,
4) and (2 ,
4) block matrices in A and B , which allows us to perform rootexchange. Repeating this process with all rows up to and including the r -th row, we obtainthe integral (5.8) with l = ( r + 1) / u ′ α,n + kr − lα (0 , Z )with l = ( r + 1) /
2. Notice that this subgroup is realized as the block matrices of size α whichare in position (( r + 1) / , ( r − /
2) in D . With this invariance, we can then proceed withroot exchange of the blocks of C and D with the corresponding blocks of A and B .We conclude from this root exchange that the integral (5.21) is zero for all choices of dataif the integral(5.23) Z ϕ ( κr ) ( i ( g ) θ (2) ,ψ n ( k − α ) ( l ( u ) ι (1 , g )) ψ U β,r ,n ( u ) ψ U α,r − ,n + βr ( u ) θ ( r )2 n + k ( r − ( u ′ α,n + βr ( Y ,r , Z ) u I α ( r − u I α ( r − ι (1 , g )) du du dY ,r dZ dg is zero for all choices of data. Here u and g are integrated as in (5.21), u is integratedover the quotient [ U α,r − ,n + βr ], and Z is integrated over [Mat α ]. Also, Y ,r is integratedover [Mat α × β ], where the notation Y ,r is explained as follows. Recall that u ′ α,n + βr ( Y, Z )was defined in (2.2). Here we have Y ∈ Mat α × n + βr ) . Write Y = (cid:0) Y Y Y (cid:1) , where Y , Y ∈ Mat α × βr and Y ∈ Mat α × n , and let Y = (cid:0) Y , Y , . . . Y ,r (cid:1) Y ,i ∈ Mat α × β ; 1 ≤ i ≤ r Then in (5.23) instead of writing Y = (cid:0) α × n + β ( r − Y ,r (cid:1) we write Y ,r for short.Next we expand the integrand in (5.23) along the group Y = (cid:0) . . . Y ,r − (cid:1) where Y ,r − ∈ [Mat α × β ]. The nontrivial characters in this expansion correspond to matricesof size β × α of fixed positive rank. We will prove that all nontrivial terms are zero. Considerthe group GL α ( F ) × GL β ( F ) embedded in Sp n + k ( r − ( F ) by( h , h ) → diag( h , . . . , h , h . . . , h , I n , h ∗ , . . . , h ∗ , h ∗ , . . . , h ∗ ) , where h ∈ GL α ( F ) appears r − h ∈ GL β ( F ) appears r times. This groupmay be used to collect terms of this expansion in the usual way. As representatives with espect to it, we choose the characters ψ A ( Y ,r − ) = ψ (tr( Y ,r − A )) A = (cid:18) I m (cid:19) ∈ Mat β × α . The contribution to the integral (5.23) from a nontrivial orbit is(5.24) Z ϕ ( κr ) ( i ( g )) θ (2) ,ψ n ( k − α ) ( l ( u ) ι (1 , g )) ψ U β,r ,n ( u ) ψ U α,r − ,n + βr ( u ) ψ A ( Y ,r − ) θ ( r )2 n + k ( r − ( u ′ α,n + βr ( Y ,r − , Y ,r , Z ) u I α ( r − u I α ( r − ι (1 , g )) du du dY ,r − dZ dg. To prove that this integral is zero, let w be a Weyl element of Sp n + k ( r − ( F ) which hasentry 1 at positions ( i, α ( i −
1) + 1), 1 ≤ i ≤ r −
1, and at positions ( r, ( α + β )( r −
1) +2( n − β ) + 1) and ( r + 1 , ( α + β )( r −
1) + 2 n − β + 1). (We do not specify it in rows r + 2to n + kr .) Using the automorphicity of θ ( r )2 n + k ( r − we can conjugate the argument of thisfunction by w . After doing so, we obtain the integral (5.3) with m = r as inner integration,and then from Lemma 4 it follows that (5.24) is zero for all choices of data. We concludethat the only nonzero contribution to the integral (5.23) from the expansion along Y ,r − isfrom the constant term.Continuing this process, next with Y ,r − , we obtain by induction that (5.23) is equal tothe integral(5.25) Z ϕ ( κr ) ( i ( g )) θ (2) ,ψ n ( k − α ) ( l ( u ) ι (1 , g )) ψ U β,r ,n ( u ) ψ U α,r − ,n + βr ( u ) θ ( r )2 n + k ( r − ( u ′ α,n + βr ( Y , Z ) u I α ( r − u I α ( r − ι (1 , g )) du du dY dZ dg, where Y is integrated over [Mat α × βr ].Next, we expand (5.25) along Y (as defined above, following (5.23)) over the quotient[Mat α × n ]. We will show that all nontrivial characters contribute zero to this expansion.Note that the group GL α ( F ) × Sp n ( F ), embedded in Sp n + k ( r − ( F ) by the map( h, g ) → diag( h, . . . , h, I βr , g, I βr , h ∗ , . . . , h ∗ ) h ∈ GL α ( F ); g ∈ Sp n ( F ) , acts, so we may consider the characters modulo this action. There are two types of repre-sentatives for the nontrivial orbits, as follows.The first is given by characters of the form ψ A ( Y ) = ψ (tr( Y A )) A = ∗ n − × n − × ∗ ∗ ∈ Mat n × α ( F ) . These appear in the expansion only if α ≥
2. Let w denote a Weyl element of Sp n + k ( r − ( F )with entry 1 in positions ( i, α ( i −
1) + 1) and ( r + i, ( α + β )( r −
1) + 2 n + iα −
1) for1 ≤ i ≤ r −
1, and in position ( r, α ( r −
1) + βr + 1). Then, arguing as above with (5.24),by conjugating by this w , we obtain the integral (5.3) with m = 2 r − otice that since α ≥
2, then k ≥
4, and we do have r ≤ r − ≤ n + kr −
1. ApplyingLemma 4, we see that the contribution to the expansion from these representatives is zero.The second type of representative is given by the characters(5.26) ψ A ( Y ) = ψ (tr( Y A )) A = (cid:18) I a (cid:19) ∈ Mat n × α ( F ); 1 ≤ a ≤ n, α. (If a > n then the orbit is represented by a character already considered above.) Thestabilizer inside Sp n contains the unipotent radical of the maximal parabolic subgroup of Sp n whose Levi part is GL a × Sp n − a ) . We denote this unipotent group by R a . It isembedded inside Sp n + k ( r − as all matrices of the form(5.27) I kr I a B CI n − a ) B ∗ I a I kr B ∈ Mat a × n − a ) ; C ∈ Mat a × a . The claim is that, as a function of g ∈ Sp n ( A ), the integral(5.28) Z θ (2) ,ψ n ( k − α ) ( l ( u ) ι (1 , g )) ψ U β,r ,n ( u ) ψ U α,r − ,n + βr ( u ) ψ A ( Y ) θ ( r )2 n + k ( r − ( u ′ α,n + βr ( Y, Z ) u I α ( r − u I α ( r − ι (1 , g )) du du dY dZ is left invariant under R a ( A ). Here Y runs over all matrices of the form (cid:0) Y Y (cid:1) with Y integrated over [Mat α × n ] and Y integrated as in (5.25). The character ψ A ( Y ) is the trivialextension of ψ A ( Y ) defined above.To prove the claim we first unfold the theta series θ (2) ,ψ n ( k − α ) , similarly to (5.15). Aftercollapsing summation and integration, we obtain(5.29) Z X ξ ∈ F m ω ψ ( l ( u ) ι (1 , g )) φ (0 , ξ ) ψ U β,r ,n ( u ) ψ U α,r − ,n + βr ( u ) ψ A ( Y ) θ ( r )2 n + k ( r − ( u ′ α,n + βr ( Y, Z ) u I α ( r − u I α ( r − ι (1 , g )) du du dY dZ. Here m = 0 if k is even, and m = n − a if k is odd. The integration in u is over the quotient U ′ β,r ,n ( F ) \ U β,r ,n ( A ) where U ′ β,r ,n is a certain subgroup of U β,r ,n (whose definition we omitas it plays no role in the sequel). It follows from the action of the Weil representation that thefunction P ξ ∈ F m ω ψ ( ι (1 , g )) φ (0 , ξ ) is left invariant under R a ( A ). After moving the matrix r a ∈ R a ( A ) to the right and changing variables, it is enough to prove that the function of g (5.30) Z θ ( r )2 n + k ( r − ( u ′ α,n + βr ( Y, Z ) u ι (1 , g )) ψ U α,r − ,n + βr ( u ) ψ A ( Y ) du dY dZ is left invariant under r a ∈ R a ( A ). Here, the variables u , Y and Z are integrated as in(5.29). o prove this invariance, we argue as in the proof of Lemma 5. First expand (5.30) alongthe abelian group Mat a × a , embedded in Sp n + k ( r − as the group of all matrices of the form(5.27) with B = 0. Next observe that only the trivial character contributes. Indeed, arguingsimilarly to the proof of Lemma 5, we obtain as inner integration either a Fourier coefficientwhich corresponds to a unipotent orbit which is not related to O (Θ ( r )2 n + k ( r − ), or an integralof the form of (5.3) with r ≤ m . Then expand along the group Mat a × n − a ) embedded inside Sp n + k ( r − as all matrices of the form (5.27) with C = 0. Similar arguments imply that onlythe constant term gives a non-zero contribution. From this it follows that integral (5.30),and hence integral (5.28), is left invariant under r a ∈ R a ( A ). Using this invariance propertyin the expansion of (5.25) along Y , when we consider the contribution from the characters(5.26), we obtain (after measure factorization) the integral Z [ R a ] ϕ ( κr ) ( i ( r a ) i ( g )) dr a as inner integration. By cuspidality this integral is zero for all choices of data.We deduce that the integral (5.25) is equal to(5.31) Z ϕ ( κr ) ( i ( g )) θ (2) ,ψ n ( k − α ) ( l ( u ) ι (1 , g )) ψ U β,r ,n ( u ) ψ U α,r − ,n + βr ( u ) θ ( r )2 n + k ( r − ( u ′ α,n + βr ( Y , Y , Z ) u I α ( r − u I α ( r − ι (1 , g )) du du dY dY dZ dg. The final expansion we need to consider is the expansion of (5.31) along the quotient Y ∈ [Mat α × βr ]. Here Y is embedded inside Sp n + k ( r − as the group of all matrices u ′ α,n + βr ( Y, Y = (cid:0) Y (cid:1) (with Y as described following (5.23)). Similarly to the expansion of(5.23) along the subgroup Y , we see that all nontrivial characters for Y give zero. Thus(5.31) is equal to(5.32) Z ϕ ( κr ) ( i ( g )) θ (2) ,ψ n ( k − α ) ( l ( u ) ι (1 , g )) ψ U β,r ,n ( u ) ψ U α,r − ,n + βr ( u ) θ ( r )2 n + k ( r − ( u ′ α,n + βr ( Y, Z ) u I α ( r − u I α ( r − ι (1 , g )) du du dY dZ dg where Y is now integrated over [Mat α × n + βr ].To complete the proof of the Theorem we need to prove that (5.32) is zero for all choicesof data. Recall that u is integrated over [ U α,r − ,n + βr ]. Combining this integration with theintegration over u ′ α,n + βr ( Y, Z ), we obtain as inner integration the constant term of the func-tion θ ( r )2 n + k ( r − along the unipotent radical of the maximal parabolic subgroup of Sp n + k ( r − whose Levi part is GL α ( r − × Sp n + β ( r − . This means that we can use Proposition 1 in[F-G2] to obtain as inner integration a function which is realized in the space of the repre-sentation σ ( r ) n,β = σ ( r ) n,k − α . Indeed, this follows since u in (5.32) is integrated over [ U β,r ,n ]. Byour assumption this representation is zero. This completes the proof of the Theorem. (cid:3) In fact it follows from the above proof that we have the following Corollary. orollary 1. Suppose that the representation σ ( r ) n,k is zero. Then, for all ≤ m ≤ [ k/ , therepresentation σ ( r ) n,k − m is zero.Proof. Suppose that σ ( r ) n,k is zero. Then for 1 ≤ α ≤ k the constant term of this representationalong the unipotent subgroup N α of SO k is zero for all choices of data. (The group N α wasdefined before (5.13).) In Theorem 2 we proved that if the representation σ ( r ) n,k − α is zero,then the constant term given by (5.13) is zero for all choices of data. In fact the converseis also true. The key point is that the process of root exchange shows that one expressionvanishes for all choices of data if and only if another does (see [G-R-S4], Corollary 7.1).Accordingly, the steps of the above proof may be reversed. We omit the detail. (cid:3) The Unramified Correspondence
Our goal in this section is to establish that the theta correspondence developed here isfunctorial on unramified principal series. In this section we let F be a nonarchimedean localfield containing µ r , and to simplify the notation we write a group or covering group for its F -points (so we write Sp n instead of Sp n ( F ), etc.). Recall that given an unramified characterof the torus of a symplectic or orthogonal group, one may define its associated principal seriesby parabolic induction from the Borel subgroup. One may do this for covering groups as well,but this requires a two-step process. Let G be the ( F -points) of one of the groups consideredabove and B = T U its standard Borel subgroup, with T be its maximal split torus. Let e G be one of the covering groups under consideration here, and if H is any subgroup of G let e H denote its inverse image in e G . Then e T is generally not abelian, but it is a two-step nilpotentgroup. If χ is a genuine character of the center Z ( e T ) of e T , then genuine principal seriesrepresentations are constructed by first extending χ to a character of a maximal abeliangroup A of e T containing Z ( e T ), next inducing this extension from A to e T , then extendingtrivially on N to obtain a representation of e B , and finally taking the normalized inductionfrom e B to e G . By an analogue of the Stone-Von Neumann Theorem, this representationis determined by its central character χ , and we write the representation π ( χ ). For moredetails see for example [McN] (analogously to the general linear group, in [K-P], Section I.1)or [Gao2]. For characters in general position these induced representations are irreducible.Let χ be the unramified character of the maximal torus T of Sp n given by χ (cid:0) diag( t , . . . , t n , t − n , . . . , t − ) (cid:1) = n Y i =1 χ i ( t i ) , where the χ i are unramified quasicharacters of F × . This character determines a genuinecharacter χ of Z ( e T ), and we form the unramified principal series π ( κr ) Sp n ( χ ) as outlined above.If κ = 2 then this requires a Weil factor, as in [B-F-H], equation (1.10). The functions inthis space are genuine with respect to the character ǫ ′ specified in Section 3. Similarly, let ξ be an unramified character of the maximal torus of SO k , ξ (cid:0) diag( t , . . . , t k , I k − k , t − k , . . . , t − ) (cid:1) = k Y i =1 ξ i ( t i ) , here k = [ k/ ξ j are quasicharacters, and the maximal torus has a 1 in the middleentry if k is odd, and form the unramified principal series π ( r ) SO k ( ξ ) (for more details see[B-F-G1], Section 6). We consider the case that k = n , so that each representation hasthe same number of Satake parameters, and study the theta correspondence between theserepresentations. (One may describe matters slightly more generally using the notion of aBrylinski-Deligne extension, as in Gao [Gao1]; see for example Leslie [L], Section 3. Thisdoes not change the discussion below in any significant way.)Let ψ be an additive character of F which is trivial on the ring of integers of F but on nolarger fractional ideal, and let ( ω ψ , V ω ψ ) be the Weil representation of Sp (2)2 nk with respect to ψ .Let (Θ ( r ) , V Θ ( r ) ) be the local theta representation of Sp ( r )2 n + k ( r − . (We shall write Θ ( r )2 n + k ( r − when we want to indicate the size of the symplectic group.) Form the vector space V ω ψ ⊗ V Θ ( r ) .This space admits a representation ω ψ ⊗ Θ ( r ) of ( Sp (2)2 nk ⋊ H kn +1 ) × Sp ( r )2 n + k ( r − . We restrictthis to a representation of SO ( r ) k × Sp ( κr )2 n using the same embeddings ι , ι as in the globalintegral. More precisely, the action of ( h, g ) ∈ SO ( r ) k × Sp ( κr )2 n on a vector v ⊗ v , v ∈ V ω ψ , v ∈ V Θ ( r ) is given by( ω ψ ⊗ Θ ( r ) )( h, g ) · ( v ⊗ v ) = ω ψ ( ι (2)1 ( p (1) ( h ) , p ( κ ) ( g ))) v ⊗ Θ ( r ) ( ι ( r )2 ( h, p ( r ) ( g ))) v . Note that the subgroup { ((1 , ζ ) , (1 , ζ − )) | ζ ∈ µ r } acts trivially. Let J U k,r ,n ,ψ Uk,r ,n or, fornotational convenience, simply J U k,r ,n ,ψ be the twisted Jacquet functor with respect to thecharacter ψ U k,r ,n of U k,r ,n ⊆ Sp n + k ( r − , acting on the first factor in the tensor product bythe map l : U k,r ,n → H kn +1 and acting on the second factor by the theta representation.(This functor is defined near the end of Section 1 above.) This is the local analogue of theintegral (3.2). Similarly to the treatment of the global situation, the group SO ( r ) k × Sp ( κr )2 n with the above action stabilizes the group U k,r ,n and character, so the Jacquet module J U k,r ,n ,ψ ( ω ψ ⊗ Θ ( r ) ) also affords a representation of SO ( r ) k × Sp ( κr )2 n . LetHom SO ( r ) k × Sp ( κr )2 n ( J U k,r ,n ,ψ ( ω ψ ⊗ Θ ( r ) ) , π ( r ) SO k ( ξ ) ⊗ π ( κr ) Sp n ( χ ))denote the space of ( SO ( r ) k × Sp ( κr )2 n )-equivariant maps from J U k,r ,n ,ψ ( ω ψ ⊗ Θ ( r ) ) to π ( r ) SO k ( ξ ) ⊗ π ( κr ) Sp n ( χ ) . Then we shall show
Theorem 3.
Let k = 2 n or n + 1 and suppose that the characters χ and ξ are in generalposition. If (6.1) Hom SO ( r ) k × Sp ( κr )2 n ( J U k,r ,n ,ψ ( ω ψ ⊗ Θ ( r ) ) , π ( r ) SO k ( ξ ) ⊗ π ( κr ) Sp n ( χ )) = 0 then after applying a Weyl group element, ξ i = χ i for each i . Here we recall that the Weyl group of Sp n acts on χ by permuting the indices and byinverting each χ i ; the Weyl group of SO k acts on the ξ i , by a similar action (if k is odd) andby permutations and by inversions of an even number of quasicharacters (if k is even). Hencethe conclusion is equivalent to the assertion that the sets { χ ± i } and { ξ ± j } , which are each ofcardinality 2 n as the characters are in general position, are in bijection. Also, if k is even thenwe recall that the principal series representations of SO ( r ) k attached to ( ξ , . . . , ξ n − , ξ n ) and( ξ , . . . , ξ n − , ξ − n ) are isomorphic under the outer automorphism of SO k that is conjugation y I k/ − I k/ − . The proof of Theorem 3 requires the local version of the smallness of the representationΘ ( r )2 l . Recall that O c (Θ ( r )2 l ) is defined in Conjecture 1 above. As in Section 4, for a givenunipotent orbit O let U O denote the upper unipotent subgroup attached to this orbit. Thenthe proof of Proposition 2 gives the following result. Proposition 7.
Suppose that O is a unipotent orbit which is greater than O c (Θ ( r )2 l ) or thatis not related to O c (Θ ( r )2 l ) . Then for any character ψ O attached to O , the twisted Jacquetmodule J U O ,ψ O (Θ ( r )2 l ) = 0 . We turn to the proof of Theorem 3. We will follow, in a loose sense, the approaches ofBump and the authors [B-F-G2], Section 6, and Leslie [L], Section 10, in their proofs ofunramified correspondences for two other theta maps that use a single theta function as anintegral kernel. Below, we refer to these proofs for some details that are similar.
Proof of Theorem 3.
We shall prove this by induction on n . More precisely, suppose thatthe nonvanishing (6.1) holds for given n , k , χ , and ξ with k = 2 n or k = 2 n + 1 and thecharacters χ , ξ in general position. Then we show that up to permutation of the indices,we have χ = ξ or χ = ξ − . We also show that a similar hypothesis holds true for( n, k ) replaced by ( n − , k −
2) and ( χ, ξ ) replaced by ( χ ′ , ξ ′ ) where χ ′ = ( χ , . . . , χ n ), ξ ′ = ( ξ , . . . , ξ n ), that is, we show thatHom SO ( r ) k − × Sp ( κr )2 n − ( J U k − ,r ,n − ,ψ ( ω ψ ⊗ Θ ( r ) ) , π ( r ) SO k − ( ξ ′ ) ⊗ π ( κr ) Sp n − ( χ ′ )) = 0Then the result follows by reduction to the case n = 1, which is straightforward to confirmby the same method used below.Let P ′ ,k − denote the unipotent radical of SO k with Levi factor M ′ ,k − := GL × SO k − andunipotent radical U ′ , k − . Let P , n − denote the unipotent radical of Sp n with Levi factor M , n − := GL × Sp n − and unipotent radical U , n − . Denote the inverse images of thesegroups in their respective covering groups by a tilde. First, by transitivity of induction, werealize π ( r ) SO k ( ξ ) as parabolically induced from ξ ⊗ π ( r ) SO k ( ξ ′ ) with ξ ′ = ( ξ , . . . , ξ n ). (Note thatthe inverse images of GL and SO k − in the local covering group SO ( r ) k commute due to blockcompatibility, so the tensor product construction is straightforward.) Similarly, π ( κr ) Sp n ( χ ) isparabolically induced from the representation χ ⊗ π ( κr ) Sp n − ( χ ′ ) where χ ′ = ( χ , . . . , χ n ). Sec-ond, the Jacquet module J U ′ ,k − ( π ( r ) SO k ( ξ )) is isomorphic to the direct sum of ξ ± i ⊗ π ( r ) SO k − ( ξ ′ i, ± )where ξ ′ i, + is obtained from ξ by removing ξ i and ξ ′ i, − is obtained from ξ by by removing ξ i and inverting one of the remaining quasicharacters. (See, for example, Section 5.2 of [B-Z].)Note that the action of the Weyl group of SO k on ξ permutes these factors. Similarly theJacquet module J U , n − ( π ( κr ) Sp n ( χ )) is isomorphic to the direct sum of χ ± j ⊗ π ( κr ) Sp n − ( χ ′ j ) where χ ′ j is obtained from χ by removing χ j . It follows that the tensor product π ( κr ) Sp n ( χ ) ⊗ π ( r ) SO k ( ξ ) s a sum of the induced representationsInd SO ( r ) k × Sp ( κr )2 n e P ′ ,k − × e P ( κr )1 , n − ( ξ ± i ⊗ π ( r ) SO k − ( ξ ′ i, ± ) ⊗ χ ± j ⊗ π ( κr ) Sp n − ( χ ′ j ) δ / P ′ ,k − δ / P , n − ) . (The weaker statement that this tensor product is glued from these representations (see[B-Z]) would be sufficient below.) Since the Weyl groups permutes these factors for differentindices, we shall focus on the case ξ ± i = ξ , χ ± j = χ without loss of generality, and in thiscase we write ξ ′ , + = ξ ′ , χ ′ = χ ′ (as in the prior paragraph).Applying Frobenius reciprocity, the nonvanishing hypothesis (6.1) in Theorem 3 impliesthat, up to the action of the Weyl groups as just explained, there is a nonzero f M ′ ,k − × f M ,n − -equivariant map J U ′ ,k − J U , n − ( J U k,r ,n ,ψ ( ω ψ ⊗ Θ ( r ) )) → ξ ⊗ π ( r ) SO k − ( ξ ′ ) ⊗ χ ⊗ π ( κr ) Sp n − ( χ ′ ) δ / P ′ ,k − δ / P , n − . To keep track of the characters that arise, let
A, B denote the block matrices(6.2) A = a g ′ a − ∈ Sp n B = b h ′ b − ∈ SO k , with a, b ∈ GL , g ′ ∈ Sp n − , h ′ ∈ SO k − . Then δ / P ′ ,k − ( B ) = | b | k/ − and δ / P , n − ( A ) = | a | n .Next we apply the argument in [G-R-S4], Proposition 6.6 (see p. 129). This implies thatthe Hom space of such maps is a quotient of the space(6.3) Hom f M ′ ,k − × f M ,n − ( J V k,r ,n ,ψ ( ω ψ ⊗ Θ ( r ) ) , ξ ⊗ π ( r ) SO k − ( ξ ′ ) ⊗ χ ⊗ π ( κr ) Sp n − ( χ ′ ) δ / P ′ ,k − δ / P , n − δ )and hence this space is nonzero. Here V k,r ,n ⊆ Sp n + k ( r − is the upper unipotent subgroup V k,r ,n = ι ( U ′ ,k − , U , n − ) U ♭k,r ,n where the group U ♭k,r ,n is defined as follows. Let Mat ′ k, n denote the subgroup of Mat k, n consisting of all matrices Y = ( y i,j ) whose entries y j, ,2 ≤ j ≤ k and y k,i , 1 ≤ i ≤ n are each zero. Then U ♭k,r ,n is the subgroup of the group U k,r ,n consisting of all matrices u whose factorization (2.3) (with a = k , b = r , c = n ) has theproperty that Y is in Mat ′ k, n . The character ψ of V k,r ,n is the character ψ U k,r ,n restricted to U ♭k,r ,n and then extended trivially to ι ( U ′ ,k − , U , n − ). Also, the map l restricted to V k,r ,n gives a homomorphism from V k,r ,n (or U ♭k,r ,n ) onto the Heisenberg group H ( n − k − andthis defines the action of V k,r ,n on ω ψ . The character δ factors through the projection from f M ′ ,k − × f M ,n − to M ′ ,k − × M ,n − , and evaluated on (6.2) has two factors, a contributionof | a | k/ − | b | n due to the action of the Weil representation and a factor from the modularfunction of the quotient U k,r ,n /U ♭k,r ,n . This quotient may be identified with the group of allmatrices u ′ k,n ( Y ,
0) (see (2.2)) such that Y is an element in Mat k, n which is the complementto Mat ′ k, n in Mat k, n , so this factor is | a | − k | b | − n . Combining these, we see that the value ofthe character δ on (6.2) above is | a | − k/ | b | − n , so δ / P ′ ,k − δ / P , n − δ takes the value | ab − | − k/ n .We remark that this step is the local analogue of using the definition of the theta functionas a sum and unfolding part of the sum, a process that is used in the proof of Theorem 2;see equation (5.15) and the discussion following, in the case α = 1, β = k − nother isomorphic Jacquet module and to conclude that the corresponding Hom space isnonzero. Globally or locally, the root exchanges require an additive character, and as in theglobal case, we use the additive character ψ of V k,r ,n obtained from ψ U k,r ,n (see (3.1)) andalso of the additive character that comes from the Weil representation ω ψ .The first set of exchanges are the same as the root exchanges used in the proof of Theorem 2above, where we treated the groups L α,β introduced in (5.18). We use these same exchangesin the case α = 1, β = k −
2. The root exchanges described through (5.20) allow us toconclude that the spaceHom f M ′ ,k − × f M ,n − ( J V ” k,r ,n ,ψ ( ω ψ ⊗ Θ ( r ) ) , ξ ⊗ π ( r ) SO k − ( ξ ′ ) ⊗ χ ⊗ π ( κr ) Sp n − ( χ ′ ) δ )is nonzero. Here V ′′ k,r ,n is the unipotent subgroup of Sp n + k ( r − generated by the followingtwo types of matrices. First, the matrices with factorization (2.3) with the X i , 1 ≤ i ≤ r − Y , and Z of the form X i = ∗ ∗ ∗ ( k − × ∗ ∗ × ( k − ∗ ∈ Mat k , Y = ∗ ∗ ∗ ( k − × ∗ ∗ × (2 n − ∈ Mat k × n ,Z = ∗ ∗ ∗ ( k − × ∗ ∗ × ( k − ∗ ∈ Mat k , where the stars indicate arbitrary elements of F . Second, the block-diagonal matrices in theLevi of P k,r ,n whose entries in the first r diagonal k × k blocks are of the form (5.18) with α = 1, β = k −
2. These root exchanges also change the character δ / P ′ ,k − δ / P , n − δ above to δ whose value on (6.2) is δ / P ′ ,k − δ / P , n − δ | b | − k − ( r − k − = | ab − | − k/ n | b | − k − ( r − k − .Next we conjugate by the Weyl group element w ∈ Sp n + k ( r − which is the shortest Weylgroup element that conjugates the matrix(6.4) diag( B, . . . , B, A, B ∗ , . . . , B ∗ ) , to the matrix(6.5) diag( b, . . . , b, a, h ′ , . . . , h ′ , g ′ , ( h ′ ) ∗ , . . . . ( h ′ ) ∗ , a − , b − , . . . , b − ) . In (6.4) above, A and B are as given in (6.2), the notation B ∗ is defined in Section 2, andeach of B, B ∗ appears r times; in (6.5), each of b, b − is repeated r − h ′ , ( h ′ ) ∗ is repeated r times. After doing so, we conclude that the Hom spaceHom f M ( J V ′ ,ψ ′ ( ω ψ ⊗ Θ ( r ) ) , ξ ⊗ π ( r ) SO k − ( ξ ′ ) ⊗ χ ⊗ π ( κr ) Sp n − ( χ ′ ) δ )is nonzero, where f M is isomorphic to f M ′ ,k − × f M ,n − and ι ( f M ′ ) projects to the group ofmatrices of the form (6.5). The group V ′ = wV ′′ k,r ,n w − is closely related to the unipotentgroup that appeared in the integration in (5.21) above. More precisely, an element in V ′ may be written as a product of the form(6.6) I r C DI C ∗ I r v u v ∗ I r R IS R ∗ I r here I = I n + k ( r − − r , u ∈ U k − ,r ,n − and v ∈ V r , which is defined as follows. Let V r bethe maximal upper unipotent subgroup of GL r . Then V r is the subgroup of V r consistingof all matrices ( v i,j ) such that v r − ,r = 0. The matrices C , D , R and S are defined similarlyto the corresponding matrices C , D , A and B in integral (5.21). With these notations thecharacter ψ ′ is described as follows. First ψ ′ (diag( v, u, v ∗ )) = ψ r ( v ) ψ U k − ,r ,n − ( u )where ψ r is the Whittaker character of V r restricted to V r . Then ψ ′ is extended trivially tothe rest of V ′ .In view of this, the space we are studying is analogous to the integral (5.21), with theunipotent integration in this local context being the process of taking the twisted Jacquetmodule, except that in that integral one is integrating g over (a cover of) Sp n while here wehave the Hom space with respect to the parabolic subgroup f M ,n − which projects to Levifactor GL × Sp n − . Note that in the proof of Theorem 2 we also conjugated by a Weylgroup element, but not quite the same one we are using now, due to the difference between Sp n and GL × Sp n − . We next use many of the same steps in the proof of Theorem 2with minor modifications.First, we carry out a series of root exchanges that are the same as the ones given following(5.22) with α = 1, β = k −
2, but with an adjustment on the size of the blocks, replacing r − r . (For example, I α ( r − in (5.21) is now replaced by I r , as in (6.6).) These areperformed row by row. Specifically, we start with the second row of the matrix C as given in(6.6). Choosing the third column in the matrix R in (6.6), we perform root exchange; this isthe same as the first in the sequence of root exchanges that was carried out following (5.22),and contributes the Jacobian | b | k . We continue to make root exchanges, again following theprocedure described following (5.22), until we have completed the first r − f M ( J V ′ ,ψ ′ ( ω ψ ⊗ Θ ( r ) ) , ξ ⊗ π ( r ) SO k − ( ξ ′ ) ⊗ χ ⊗ π ( κr ) Sp n − ( χ ′ ) δ ) . Here V ′ is the subgroup of Sp n + k ( r − of matrices which have a factorization (6.6) thatsatisfies the following conditions: for C : C i,j = 0 if i = r −
1, 1 ≤ j ≤ n + ( r − k − − k or i = r , 1 ≤ j ≤ r ( k − D , D r − , = D r − , = D r, = 0; for R , R i,j = 0 if 1 ≤ j ≤ r − j = r , i > ( r − k − S = 0. Also, ψ ′ is the character of V ′ that is ψ ′ extendedtrivially to V ′ . (Note that studying this nonvanishing is the local analogue of studying thepossible nonvanishing of (5.23) with α = 1 and r − r , and this same groupand character, up to this modification, appear there.) The character δ in (6.7) is computedas follows. In carrying out these root exchanges, the character δ , evaluated on (6.2), ismultiplied by a power of | b | at each step of the root exchange process. For 1 ≤ j ≤ r , tofill in the j -th row by root exchange the adjustment on the character is | b | k ( j − , and for row r + j , 1 ≤ j ≤ r −
1, the adjustment is | b | r k +2 n − j − k − . The character δ in (6.7) isobtained by combining these adjustments with the prior character δ . (We will compute thefinal character later in the proof.)We next treat the ( r − L denote a unipotent abelian subgroup of Sp n + k ( r − suchthat LV ′ is also a unipotent group, and such that ψ ′ is a well-defined character of LV ′ whenextended trivially from V ′ to LV ′ . Also suppose that the group M normalizes this unipotent roup. Then it follows from the nonvanishing of the Hom space (6.7) that at least one ofthe spaces(6.8) Hom f M ψL ( J L,ψ L ( J V ′ ,ψ ′ ( ω ψ ⊗ Θ ( r ) )) , ξ ⊗ π ( r ) SO k − ( ξ ′ ) ⊗ χ ⊗ π ( κr ) Sp n − ( χ ′ ) δ ) . is non-zero, as ψ L runs over a set of representatives of the characters of the group L underthe action of the group M . Here f M ψ L is the stabilizer of ψ L in f M . Indeed, this follows fromthe gluing described in Bernstein and Zelevinsky [B-Z], Theorem 5.2, applied as in the workof Bump and the authors [B-F-G2], pp. 392-3. As explained there, one obtains an exactsequence from the Geometrical Lemma of [B-Z], p. 448, and then applies a Jacquet functorwhich preserves the sequence (as it is an exact functor). One term involves a Jacquet functorapplied to a compactly-induced representation, and to study the Hom space from this piece,one applies Mackey theory as extended in [B-Z].We begin to make this expansion with the root supported on one-parameter subgroup E r − , n + k ( r − − r +2 (defined before Proposition 6). In fact, this is the same procedure used toanalyze (5.23) and described in detail following (5.23). The Jacquet module with the charac-ter (the ‘non-constant term’ in the global computation) factors through the twisted Jacquetmodule corresponding to the orbit (( r + 1)1 n + k ( r − − r +1 ) for Θ ( r ) . However, this vanishes bythe smallness of the representation Θ ( r ) , Proposition 7. Let V ′ = E r − , n + k ( r − − r +2 V ′ , andlet ψ ′ denote the character of V ′ which is the trivial extension of ψ ′ . Then we deduce thatthe space (6.7) is isomorphic to(6.9) Hom f M ( J V ′ ,ψ ′ ( ω ψ ⊗ Θ ( r ) ) , ξ ⊗ π ( r ) SO k − ( ξ ′ ) ⊗ χ ⊗ π ( κr ) Sp n − ( χ ′ ) δ ) , hence nonzero.We move to the next root, with root subgroup E ′ r − , n + k ( r − − r +1 . For convenience denotethis group L , and let x ( t ) = I n + k ( r − + te ′ r − , n + k ( r − − r +1 ∈ L ( e ′ i,j is defined before Proposition 6). The group M consisting of all matrices of the form (6.5)acts on L by conjugation with two orbits; indeed if m is given by (6.5), then mx ( t ) m − = x ( abt ). Let M ⊂ M denote the stabilizer of this action; this is the subgroup consisting ofall matrices (6.5) with b = a − . Let ψ L denote a nontrivial character of L . Then, since (6.9)is not zero, we deduce that at least one of the two spaces(6.10) Hom f M ( J L,ψ L ( J V ′ ,ψ ′ ( ω ψ ⊗ Θ ( r ) )) , ξ ⊗ π ( r ) SO k − ( ξ ′ ) ⊗ χ ⊗ π ( κr ) Sp n − ( χ ′ ) δ )and(6.11) Hom f M ( J V ′ ,ψ ′ ( ω ψ ⊗ Θ ( r ) ) , ξ ⊗ π ( r ) SO k − ( ξ ′ ) ⊗ χ ⊗ π ( κr ) Sp n − ( χ ′ ) δ )is also not zero. Here V ′ = LV ′ and the character ψ ′ is obtained by extending ψ ′ triviallyto V ′ .We start with the case that the space (6.10) is not zero. (We will deal with the space(6.11) afterwards.) There the character ψ L , which appears through the twisting, allows usto do root exchanges (similar to the the treatment of (5.23) above), in order to fill in all theremaining positive root subgroups E ′ r − ,j in row r −
1, as follows. First, we use ψ L to moveroots into row r − E ′ r,j with r + r ( k − < j ≤ n + k ( r − − r and E r, n + k ( r − − r +1 that appear in V ′ in row r . We then use root exchange to exchangethe ( r − k −
2) negative roots E ′ j,r , r < j ≤ r + ( r − k −
2) of V ′ (these appear in and R ∗ of (6.6)) into row r −
1. Note that this root exchange is possible because thereis an additive character appearing from the Weil representation ω ψ . Doing these exchangesmultiplies the character δ by a factor of | b | r k +2 n − r − k − | a | ( r − k − . Thus we obtainthe nonvanishing of the Hom spaceHom f M ( J L ′ ,ψ L ′ ( J V ′′ ,ψ ′ ( ω ψ ⊗ Θ ( r ) )) , ξ ⊗ π ( r ) SO k − ( ξ ′ ) ⊗ χ ⊗ π ( κr ) Sp n − ( χ ′ ) δ ) . Here L ′ is the (abelian) group generated by L and all the positive root subgroups of theform E ′ r − ,j , r ≤ j ≤ n + k ( r − − r (note that this group includes E r − , n + k ( r − − r +2 ), ψ L ′ is ψ L extended trivially from L to L ′ , V ′′ is the subgroup of V ′ consisting of upperunipotent matrices with 0 in position ( r, j ), r < j ≤ n + k ( r − − r + 1 and δ = δ | b | r k +2 n − r − k − | a | ( r − k − . At this point we conjugate by the shortest Weyl group element, call it w ′ , that interchanges a and a − in (6.5). We then fill in the remaining positive roots in row r . To do so werepeat the same process as above. We start with the group E r, n + k ( r − − r +1 . Expanding,the twisted Jacquet module vanishes by the smallness of Θ ( r ) . Then we consider the group L ′′ /E r, n + k ( r − − r +1 where L ′′ is the group generated by E r, n + k ( r − − r +1 and the E ′ r,j with r + 1 ≤ j ≤ n + k ( r − − r . The Hom spaces with nontrivial characters at these rootseach are zero, since nonvanishing for a nontrivial character in row r would imply that therepresentation Θ ( r ) supports a functional for a unipotent orbit that has at least r + 1 in itspartition, and this would contradict Proposition 7.Putting all this together, the nonvanishing of the Hom space (6.10) implies thatHom f M ( J U k − ,r ,n − ,ψ ( ω ψ ⊗ J U GLr ,ψ Wh ( J U r, ,n + kr − r (Θ ( r ) ))) , ξ ⊗ π ( r ) SO k − ( ξ ′ ) ⊗ χ − ⊗ π ( κr ) Sp n − ( χ ′ ) δ )is nonzero. Here f M = w ′ f M ( w ′ ) − . The projection of f M to Sp n + k ( r − isdiag( a − , . . . , a − , h ′ , . . . , h ′ , g ′ , ( h ′ ) ∗ , . . . . ( h ′ ) ∗ , a, . . . , a )where a − and a are each repeated r times and h ′ , ( h ′ ) ∗ are each repeated r times. TheJacquet functor applied to Θ ( r ) := Θ ( r )2 n + k ( r − is the untwisted Jacquet functor with respectto U r, ,n + kr − r , the unipotent radical of the parabolic with Levi factor GL r × Sp n + k ( r − − r .By Proposition 1, this functor gives the module Θ ( r ) GL r ⊗ Θ ( r )2 n +2 k ( r − − r . Here Θ ( r ) GL r denotesthe local theta representation for GL ( r ) r ( F ). The Jacquet functor J U GLr ,ψ Wh is the twistedJacquet functor with respect to the Whittaker character of GL r , and is applied to the firstcomponent of this tensor product.However, the theta representation on the r -fold cover of GL r is generic by [K-P], and itscentral character is computed there. Thus the action of GL (more precisely, the subgroupof central elements in the metaplectic group) is as follows. If B is the Borel of Sp n + k ( r − then we obtain a total contribution from this step of δ r − r B a − I r I n + k ( r − − r aI r = | a | − r (2 n +( k − r − . The character δ is given on (6.2) with b = a − by | a | − raised to the power obtained bycollecting all the terms arising from the unfolding and the root exchanges, each of which is oted above. This power is k + ( r − k −
2) + r − X j =1 jk + r ( r k + 2 n −
2) + r − X j =1 j ( k − − ( r − k − r (2 n + ( k − r − . This power of | a | − thus exactly matches the corresponding contribution from the Whittakercoefficient of Θ ( r ) GL r .We conclude that if the Hom space (6.10) is nonzero, then χ = ξ and moreover the spaceHom SO ( r ) k − × Sp ( κr )2 n − ( J U k − ,r ,n − ,ψ ( ω ψ ⊗ Θ ( r )2 n − k − r − )) , π ( r ) SO k − ( ξ ′ ) ⊗ π ( κr ) Sp n − ( χ ′ ))is nonzero. This is the same nonvanishing as in (6.1) but with ( n, k ) replaced by ( n − , k − χ, ξ ) replaced by ( χ ′ , ξ ′ ). Thus in this case we are done by induction.We must also analyze the situation that it is the Hom space (6.11) which is non-zero.In this case, we continue to expand along the roots of row r − h ′ and the final r entries of the matrix (6.5). We first expand along the root spaces E ′ r − ,j with r < j ≤ n + k ( r − − r − ( k − SO k − and Sp k − which act on the characters in the Fourier expansion by conjugation (see following (6.8)).Thus we expand using the abelian groups consisting of k − E ′ r,j at a timefor the roots whose root spaces include an entry in the r -th row above each h ′ and h and2 n − g ′ . It follows by using the smallness ofΘ ( r ) or general position (when a copy of GL , restricted to r -th powers, acts on the moduleunder consideration by a fixed character) that the contributions coming from each nontrivialcharacter in this expansion vanish. We conclude that the spaceHom f M ( J V ′ ,ψ ′ ( ω ψ ⊗ Θ ( r ) ) , ξ ⊗ π ( r ) SO k − ( ξ ′ ) ⊗ χ ⊗ π ( κr ) Sp n − ( χ ′ ) δ )is non-zero, where V ′ is the group generated by V ′ and the root subgroups E ′ r − ,j with r < j ≤ n + k ( r − − r + 1, and the character ψ ′ is ψ ′ extended trivially to V ′ .We now do the expansion along the root subgroup E ′ r − ,r . This is again glued from twoterms, but the situation here is different. First, there is a constant term at this root. In thiscase, the Jacquet module arising from the expansion factors through J U r − , , n +( k − r − (Θ ( r ) ).By Proposition 1, this gives the representation Θ ( r ) GL r − ⊗ Θ ( r )2 n +( k − r − . However, here the GL in position b in (6.5) (restricted to r -th powers) acts by a fixed character. Since we areassuming that χ and ξ are in general position, the Hom space from this term is zero. Weconclude that it is the non-constant piece that must be non-zero.To analyze the non-constant piece, i.e. the contribution from a nontrivial character ψ r − ,r of the group E ′ r − ,r , we must restrict to the stabilizer in f M of this character. If x r − ,r ( t ) = I n + k ( r − + te ′ r − ,r and m is given by (6.5) then mx r − ,r ( t ) m − = x r − ,r ( ba − t ). Thus thischaracter is stabilized by all matrices (6.5) with a = b . One argues similarly to the above tosee that this Hom space is nonzero only if χ = ξ − and ifHom SO ( r ) k − × Sp ( κr )2 n − ( J U k − ,r ,n − ,ψ ( ω ψ ⊗ Θ ( r )2 n − k − r − )) , π ( r ) SO k − ( ξ ′ ) ⊗ π ( κr ) Sp n − ( χ ′ )) = (0) . Then we are done by induction.This concludes the proof of Theorem 3. (cid:3) eferences [B-L-S] Banks, W.; Levy, J.; Sepanski, M. R.: Block-compatible metaplectic cocycles. J. Reine Angew.Math. (1999), 131–163.[B-Z] Bernstein, I.N.; Zelevinsky, A.V.: Induced representations of reductive p -adic groups, I. Ann.scient. ´Ec. Norm. Sup. (1977), no. 4, 441–472.[B-F-G1] Bump, D.; Friedberg, S.; Ginzburg, D.: Small representations for odd orthogonal groups. Intern.Math Res. Notices (IMRN) (2003), no. 25, 1363–1393.[B-F-G2] Bump, D.; Friedberg, S.; Ginzburg, D.: Lifting automorphic representations on the double coversof orthogonal groups. Duke Math. J. (2006), no. 2, 363–396.[B-F-H] Bump, D.; Friedberg, S.; Hoffstein, J.: p -adic Whittaker functions on the metaplectic group. DukeMath. J. (1991), no. 2, 379–397.[B-D] Brylinski, J-L; Deligne, P.: Central extensions of reductive groups by K . Publ. Math. Inst.Hautes ´Etudes Sci. No. 94 (2001), 5–85.[C1] Cai, Y.: Fourier coefficients for theta representations on covers of general linear groups. Trans.Amer. Math. Soc. (2019), no. 11, 7585–7626.[C2] Cai, Y.: Twisted doubling integrals for Brylinski-Deligne extensions of classical groups.arXiv:1912.08068.[CFGK1] Cai, Y.; Friedberg, S.; Ginzburg, D.; Kaplan, E.: Doubling constructions and tensor productL-functions: the linear case. Invent. Math. (2019), no. 3, 985–1068.[CFGK2] Cai, Y.; Friedberg, S.; Ginzburg, D.; Kaplan, E.: Doubling constructions for covering groups andtensor product L-functions. arXiv:1601.08240.[C-M] Collingwood; D. H., McGovern, W. M.: Nilpotent orbits in semisimple Lie algebras. Van NostrandReinhold Mathematics Series. Van Nostrand Reinhold Co., New York (1993).[F-G1] Friedberg, S.; Ginzburg, D.: Criteria for the existence of cuspidal theta representations. Res.Number Theory 2 (2016), Art. 16, 16 pp.[F-G2] Friedberg, S.; Ginzburg, D.: Theta functions on covers of symplectic groups. In: AutomorphicForms, L-functions and Number Theory, a special issue of BIMS in honor of Prof. FreydoonShahidi’s 70th birthday. Bull. Iranian Math. Soc. (4) (2017), no. 4, 89–116.[F-G3] Friedberg, S.; Ginzburg, D.: Descent and theta functions for metaplectic groups. J. Eur. Math.Soc. 20 (2018), 1913–1957.[F-G4] Dimensions of automorphic representations, L -functions and liftings. To appear in Trace Formulas,a volume in the series Springer Symposia.[F-G5] Friedberg, S.; Ginzburg, D.: The first occurrence for the generalized classical theta lift, in prepa-ration.[Gao1] Gao, F.: The Langlands-Shahidi L -functions for Brylinski-Deligne extensions. Amer. J. Math. (2018), no. 1, 83–137.[Gao2] Gao, F.: Distinguished theta representations for certain covering groups. Pacific J. Math. 290(2017), no. 2, 333–379.[G-T] Gao, F.; Tsai, W.-Y.: On the wavefront sets associated with theta representations.arXiv:2008.03630.[G] Ginzburg, D.: Certain conjectures relating unipotent orbits to automorphic representations. IsraelJ. Math. (2006), 323–355.[G-R-S1] Ginzburg, D.; Rallis, S.; Soudry, D.: L-functions for symplectic groups. Bull. Soc. Math. France (1998), no. 2, 181–244.[G-R-S2] Ginzburg, D.; Rallis, S.; Soudry, D.: On a correspondence between cuspidal representations of GL n and f Sp n . J. Amer. Math. Soc. 12 (1999), no. 2, 243–266.[G-R-S3] Ginzburg, D.; Rallis, S.; Soudry, D.: On Fourier coefficients of automorphic forms of symplecticgroups. Manuscripta Math. (2003), no. 1, 1–16.[G-R-S4] Ginzburg, D.; Rallis, S.; Soudry, D.: The descent map from automorphic representations of GL ( n )to classical groups. World Scientific Publishing Co. Pte. Ltd., Hackensack, NJ (2011). Ho] Howe, R.: θ -series and invariant theory. Automorphic forms, representations and L-functions(Proc. Sympos. Pure Math., Oregon State Univ., Corvallis, Ore., 1977), Part 1, pp. 275–285,Proc. Sympos. Pure Math., XXXIII, Amer. Math. Soc., Providence, R.I., 1979.[Ik1] Ikeda, T.: On the theory of Jacobi forms and Fourier-Jacobi coefficients of Eisenstein series. J.Math. Kyoto Univ. (1994), no. 3, 615–636.[Ik2] Ikeda, T.: On the residue of the Eisenstein series and the Siegel-Weil formula. Compositio Math. (1996), no. 2, 183–218.[J-L] Jiang, D.; Liu, B.: On special unipotent orbits and Fourier coefficients for automorphic forms onsymplectic groups. J. Number Theory (2015), 343—389.[Kap] Kaplan, E.: Doubling Constructions and Tensor Product L-Functions: coverings of the symplecticgroup. arXiv:1902.00880.[K-P] Kazhdan, D. A.; Patterson, S. J.: Metaplectic forms. Inst. Hautes ´Etudes Sci. Publ. Math., No.59 (1984), 35–142.[Ku] Kubota, T.: Some results concerning reciprocity and functional analysis. Actes du Congr`es Inter-national des Math´ematiciens (Nice, 1970), Tome 1, pp. 395–399. Gauthier-Villars, Paris, 1971.[Ku1] Kudla, S.: Splitting metaplectic covers of dual reductive pairs. Israel J. Math. ∼ skudla/ssk.research.html.[L] Leslie, S.: A Generalized Theta lifting, CAP representations, and Arthur parameters. Trans.Amer. Math. Soc. (2019), no. 7, 5069–5121.[Mat] Matsumoto, H.: Sur les sous-groupes arithm´etiques des groupes semi-simples d´eploy´es. Ann. Sci.´Ecole Norm. Sup. (4) 2 (1969), 1–62.[McN] McNamara, P.: Principal series representations of metaplectic groups over local fields. In:Multiple Dirichlet series, L-functions and automorphic forms, 299–327, Progr. Math., 300,Birkh¨auser/Springer, New York, 2012.[M-W] Mœglin, C.; Waldspurger, J.-L.: Spectral decomposition and Eisenstein series. Une paraphrasede l’´Ecriture [A paraphrase of Scripture]. Cambridge Tracts in Mathematics, 113. CambridgeUniversity Press, Cambridge, 1995[Mo] Moore, C. C.: Group extensions of p -adic and adelic linear groups. Inst. Hautes ´Etudes Sci. Publ.Math., No. 35 (1968), 5–70.[O-S] Offen, O.; Sayag, E.: Global mixed periods and local Klyachko models for the general lineargroup. Int. Math. Res. Not. IMRN , no. 1, Art. ID rnm 136, 25 pp.[Ra1] Rallis, S.: Langlands’ functoriality and the Weil representation. Amer. J. Math. (1982), no.3, 469–515.[Ra2] Rallis, S.: L -functions and the oscillator representation. Lecture Notes in Mathematics, 1245.Springer-Verlag, Berlin, 1987.[Rao] Ranga Rao, R.: On some explicit formulas in the theory of Weil representation. Pacific J. Math. (1993), no. 2, 335–371.[Ro] Roberts, B. Nonvanishing of global theta lifts from orthogonal groups. J. Ramanujan Math. Soc. (1999), no. 2, 131–194.[Su1] Suzuki, T.: Distinguished representations of metaplectic groups. Amer. J. Math. (1998), no.4, 723–755.[Su2] Suzuki, T.: On the Fourier coefficients of metaplectic forms. Ryukyu Math. J. (2012), 21–106.[Sw] Sweet, W. J. Jr.: The metaplectic case of the Weil-Siegel formula. Thesis, Univ. of Maryland,1990.[Tak] Takeda, S.: The twisted symmetric square L -function of GL ( r ). Duke Math. J. (2014), no.1, 175–266.[We] Weil, A.: Sur certains groupes d’op´erateurs unitaires. Acta Math. (1964), 143–211.[W] Weissman, M. H.: L-groups and parameters for covering groups. In: L-groups and the Langlandsprogram for covering groups. Ast´erisque 2018, no. 398, 33–186. riedberg: Department of Mathematics, Boston College, Chestnut Hill, MA 02467-3806,USA E-mail address : [email protected] Ginzburg: School of Mathematical Sciences, Tel Aviv University, Ramat Aviv, Tel Aviv6997801, Israel
E-mail address : [email protected]@post.tau.ac.il