aa r X i v : . [ m a t h . R T ] O c t ON REPRESENTATIONS OF REAL JACOBI GROUPS
BINYONG SUN
Abstract.
We consider a category of continuous Hilbert space representationsand a category of smooth Fr´echet representations, of a real Jacobi group G . ByMackey’s theory, they are respectively equivalent to certain categories of represen-tations of a real reductive group e L . Within these categories, we show that the twofunctors of taking smooth vectors for G , and for e L , are consistent with each other.By using Casselman-Wallach’s theory of smooth representations of real reductivegroups, we define matrix coefficients for distributional vectors of certain represen-tations of G . We also formulate Gelfand-Kazhdan criteria for Jacobi groups whichcould be used to prove the multiplicity one theorem for Fourier-Jacobi models. Introduction
By a (complex) representation of a Lie group, we mean a continuous linear actionof it on a complete locally convex space. In this paper, a locally convex space meansa complex topological vector space which is Hausdorff and locally convex. When noconfusion is possible, we do not distinguish a representation with its underlying space.Representations of reductive groups are studied intensively in the literature. Theseare the most interesting groups from the point of view of the Langlands program.But there is another family of groups which show up quite often in number theory,namely, Jacobi groups.We work in the setting of Nash groups. By a Nash group, we mean a group whichis simultaneously a Nash manifold so that all group operations (the multiplicationand the inversion) are Nash maps. The reader is referred to [Sh1, Sh2] for details onNash manifolds and Nash groups. Let G be a Nash group. A finite dimensional realrepresentation E of G is said to a Nash representation if the action map G × E → E is Nash. A Nash group is said to be almost linear if it admits a Nash representationwith finite kernel. It is said to be unipotent if it admits a faithful Nash representationso that all group elements act as unipotent operators. Every unipotent Nash groupis connected and simply connected. As in the case of linear algebraic groups, everyalmost linear Nash group has a unipotent radical, namely, a largest unipotent normal Mathematics Subject Classification.
Key words and phrases.
Jacobi groups, Heisenberg groups, Casselman-Wallach representations,matrix coefficients, Gelfand-Kazhdan criteria.
Nash subgroup of it. (Recall that every Nash subgroup is automatically closed.) Analmost linear Nash group is said to be reductive if its unipotent radical is trivial.Assume that G is almost linear, and denote by H its unipotent radical. The centerof G is also an almost linear Nash group. Its unipotent radical Z is called theunipotent center of G . This equals to the intersection of H with the center of G . If e G → G is a finite cover of Lie groups. Then e G is uniquely a Nash group so that thecovering map is Nash. In this case, the unipotent radical (and the unipotent center)of G and e G are canonically identified.As usual, we use the corresponding lower case German letter to denote the Liealgebra of a Lie group. We say that the almost linear Nash group G is a real Jacobigroup if [ h , h ] ⊂ z and the map [ , ] : h / z × h / z → z is non-degenerate in the sense that the alternating form φ ◦ [ , ] : h / z × h / z → R is non-degenerate for some linear functional φ : z → R . By definition, every reductiveNash group is a real Jacobi group. It is easy to see that all real Jacobi groups areunimodular.Now assume that G is a real Jacobi group. Fix a unitary character ψ on Z whichis generic in the sense that the alternating form(1) dψ √− ◦ [ , ] : h / z × h / z → R is non-degenerate, where dψ denotes the differential of ψ . A representation of G issaid to be a ψ -representation if Z acts through the character ψ .Among all representations, unitary ones are most important for many applications.But non-unitary representations also occur naturally even in the study of unitaryones. In general, we will consider ψ -representations of G on Hilbert spaces so that H acts by unitary operators. Denote by H mod G,ψ the category of these representations.Morphisms in this category are G -intertwining continuous linear maps (which mayor may not preserve the inner products).Smooth representations are also important for many purposes. In this paper, weare very much concerned with smooth Fr´echet ψ -representations of G of moderategrowth. (A representation is said to be Fr´echet if its underlying space is. See Section2.2 for the notion of smooth representations, and see Section 2.3 for the moderategrowth condition.) Following F. du Cloux ([du4]), denote by J mod G,ψ the categoryof these representations.Given any representation V in H mod G,ψ , it turns out (cf. Corollary 4.3) that itssmooth vectors V ∞ form a representation in J mod G,ψ (see Section 2.2 for the notion
N REPRESENTATIONS OF REAL JACOBI GROUPS 3 of smooth vectors). Therefore we get a functor (the smoothing functor)(2) ( · ) ∞ : H mod G,ψ → J mod
G,ψ . We are aimed to show that the smoothing functor (2) can be identified with thesmoothing functor for a reductive group.To be more precise, use the alternating form (1), we form the real symplecticgroup Sp( h / z ) and its metaplectic double cover f Sp( h / z ). The adjoint action inducesa homomorphism L := G/H → Sp( h / z ) . Define the fibre product e L := L × Sp( h / z ) f Sp( h / z ) . This is a double cover of L . All the groups Sp( h / z ), f Sp( h / z ), L and e L are canonicallyreductive Nash groups.A representation of e L is said to be genuine if the nontrivial element in the kernel ofthe covering map e L → L acts as the scalar multiplication by −
1. This terminologyapplies to other double covers of groups. Denote by H mod e L, gen the category ofgenuine representations of e L on Hilbert spaces, and by J mod e L, gen the category ofgenuine smooth Fr´echet representations of e L of moderate growth. As in the case ofJacobi groups, we still have the smoothing functor:(3) ( · ) ∞ : H mod e L, gen → J mod e L, gen . Put e G := G × L e L. This is a double cover of G so that e G/H = e L . In order to relate ψ -representationsof G to genuine representations of e L , we fix a unitary oscillator representation ω h ψ of e G corresponding to ψ : ω h ψ is a genuine unitary ψ -representation of e G which remainsirreducible when restricted to H . Such a representation always exists (see Section4), and all others are unitarily isomorphic to twists of ω h ψ by unitary characters of L . Denote by ω ψ the smoothing of ω h ψ . We construct in Section 4 four functors:Hom H ( ω h ψ , · ) : H mod G,ψ → H mod e L, gen , (4) ω h ψ b ⊗ h · : H mod e L, gen → H mod G,ψ , (5) Hom H ( ω ψ , · ) : J mod G,ψ → J mod e L, gen , (6) ω ψ b ⊗ · : J mod e L, gen → J mod G,ψ . (7)The usual meanings of the topological tensor products “ b ⊗ h ” and “ b ⊗ ” will be ex-plained in Section 2.1.One purpose of this paper is to clarify the following B. SUN
Theorem A.
The functors (4) and (5) are inverse to each other, the functors (6) and (7) are inverse to each other, and the diagrams H mod G,ψ ( · ) ∞ −−−→ J mod G,ψ
Hom H ( ω h ψ , · ) y y Hom H ( ω ψ , · ) H mod e L, gen ( · ) ∞ −−−→ J mod e L, gen and H mod G,ψ ( · ) ∞ −−−→ J mod G,ψω h ψ b ⊗ h x x ω ψ b ⊗ H mod e L, gen ( · ) ∞ −−−→ J mod e L, gen commute. More precisely, Theorem A says that we have the following natural identificationsof representations: ω h ψ b ⊗ h Hom H ( ω h ψ , V h ) = V h , Hom H ( ω h ψ , ω h ψ b ⊗ h E h ) = E h ,ω ψ b ⊗ Hom H ( ω ψ , V ) = V, Hom H ( ω ψ , ω ψ b ⊗ E ) = E, Hom H ( ω h ψ , V h ) ∞ = Hom H ( ω ψ , V h ∞ ) , ( ω h ψ b ⊗ h E h ) ∞ = ω ψ b ⊗ E h ∞ , for all representations V h in H mod G,ψ , E h in H mod e L, gen , V in J mod G,ψ , and E in J mod e L, gen . The first two assertions of Theorem A are in some sense well known(Mackey’s theory, cf. [du4, Proposition 4.3.9]). They are rather direct consequencesof the work of F. du Cloux ([du2, du3, du4]). The last assertion of Theorem Ageneralizes the expectation of R. Berndt in [Be, Page 185].We say that a representation of a reductive Nash group is a Casselman-Wallachrepresentation if it is Fr´echet, smooth, of moderate growth, admissible and Z-finite.Here Z is the center of the universal enveloping algebra of the complexified Liealgebra of the group. The reader may consult [Ca], [Wa, Chapter 11] and [BK] formore details about Casselman-Wallach representations.Denote by F H e L, gen the category of genuine Casselman-Wallach representations of e L . This is a full subcategory of J mod e L, gen . Denote by F H
G,ψ the full subcategory of J mod G,ψ corresponding to
F H e L, gen , under the functors (6) and (7). Objects of this N REPRESENTATIONS OF REAL JACOBI GROUPS 5 category are called Casselman-Wallach ψ -representations of G . These representationsshould be useful in the study of Jacobi forms (cf. [EZ]). It is important to note thatthe underlying spaces of all Casselman-Wallach ψ -representations are nuclear.As an application of Theorem A, we define matrix coefficients for distributionalvectors in Casselman-Wallach ψ -representations, as what follows. Denote by C ξ ( G )the space of tempered smooth functions on G , and by C − ξ ( G ) the locally convexspace of tempered generalized functions on G . The later contains the former as adense subspace (see Section 2.3 for precise definitions of these spaces).Let U be a representation in F H
G,ψ and let V be a representation in F H G, ¯ ψ whichare contragredient to each other, namely, a non-degenerate G -invariant continuousbilinear form h , i : U × V → C is given. Note that every representation in F H
G,ψ has a unique contragredient rep-resentation in
F H G, ¯ ψ (Proposition 5.3). Denote by U −∞ the strong dual of V . It isa smooth representation of G containing U as a dense subspace. Similarly, denoteby V −∞ the strong dual of U .For any u ∈ U , v ∈ V , the (usual) matrix coefficient c u ⊗ v is defined by(8) c u ⊗ v ( g ) := h gu, v i , g ∈ G. It is a function in C ξ ( G ). The following result is proved in [SZ1, Theorem 2.1] forreal reductive groups. Corollary B.
With the notation as above, the matrix coefficient map (9) U × V → C ξ ( G ) , ( u, v ) c u ⊗ v extends to a continuous bilinear map U −∞ × V −∞ → C − ξ ( G ) , and the induced G × G intertwining linear map (10) c : U −∞ b ⊗ V −∞ → C − ξ ( G ) is a topological homomorphism with closed image. Recall that a linear map φ : E → F of locally convex spaces is called a topologicalhomomorphism if the induced linear isomorphism E/ Ker( φ ) → Im( φ ) is a homeo-morphism, where E/ Ker( φ ) is equipped with the quotient topology of E , and theimage Im( φ ) is equipped with the subspace topology of F . The action of G × G on C − ξ ( G ) is given by (( g , g ) .f )( x ) := f ( g − xg ) . B. SUN
In particular, Corollary B defines characters of Casselman-Wallach ψ -representations(as tempered generalized functions on G ). It also implies that irreducible Casselman-Wallach ψ -representations are determined by their characters ([SZ1, Remark 2.2]).The following form of Gelfand-Kazhdan criteria is a rather direct consequence ofCorollary B. See [SZ1, Theorem 2.3] for a proof. Corollary C.
Let S and S be two closed subgroups of the real Jacobi group G , withcontinuous (non-necessarily unitary) characters χ i : S i → C × , i = 1 , . Assume that there is a Nash anti-automorphism σ of G such that for every f ∈ C − ξ ( G ) which is an eigenvector of U( g C ) G , the conditions f ( zx ) = ψ ( z ) f ( x ) , z ∈ Z,f ( sx ) = χ ( s ) f ( x ) , s ∈ S , and f ( xs ) = χ ( s ) − f ( x ) , s ∈ S imply that f ( x σ ) = f ( x ) . Then for every pair of irreducible representations U in F H
G,ψ and V in F H G, ¯ ψ whichare contragredient to each other, one has that dim Hom S ( U, χ ) dim Hom S ( V, χ ) ≤ . Here and henceforth, a subscript “ C ” indicates the complexification of a real Liealgebra, the universal enveloping algebra U( g C ) is identified with the algebra of leftinvariant differential operators on G , and U( g C ) G is identified with the algebra ofbi-invariant differential operators on G .Another purpose of this paper is to have the following criterion for a strong Gelfandpair, which is used in [SZ2] to prove the multiplicity one theorem for Fourier-Jacobimodels. As in the proof of [SZ1, Corollary 2.5], the criterion is implied by CorollaryC. We shall not go to the details. Corollary D.
Let G ′ be a Nash subgroup of G which is also a real Jacobi grouop. Fixa generic unitary character ψ ′ of the unipotent center Z ′ of G ′ . Assume that thereexists a Nash anti-automorphism σ of G preserving G ′ with the following property:every tempered generalized function on G which is invariant under the adjoint actionof G ′ is automatically σ -invariant. Then for every irreducible Casselman-Wallach ψ -representation U of G , and every irreducible Casselman-Wallach ψ ′ -representation U ′ of G ′ , the space of G ′ -invariant continuous bilinear functionals on U × U ′ is atmost one dimensional. N REPRESENTATIONS OF REAL JACOBI GROUPS 7
Acknowledgements: This paper is an outcome of discussions between AvrahamAizenbud and the author. In particular, the author learned Lemma 2.1 and Lemma4.4 from him. The author is very grateful to him. The work was partially supportedby NSFC grants 10801126 and 10931006.2.
Preliminaries
In this section, we fix some notations and terminologies and recall some generalresults which will be used later in this paper.2.1.
Topological tensor products.
Let E and F be two locally convex spaces.There are at least four useful locally convex topologies one can put on the algebraictensor product E ⊗ F , namely, the inductive tensor product E ⊗ i F , the projectivetensor product E ⊗ π F , the epsilon tensor product E ⊗ ǫ F and the Hilbert tensorproduct E ⊗ h F . As usual, their completions are denoted by E b ⊗ i F , E b ⊗ π F , E b ⊗ ǫ F and E b ⊗ h F , respectively. The inductive tensor product plays no role in this paper.The projective tensor product and the epsilon tensor product are more commonlyused and the reader is referred to [Th] for a concise treatment. A fundamentaltheorem of Grothedieck says that they coincide when either E or F is nuclear. Ifthis is the case, we simply write E b ⊗ F := E b ⊗ π F = E b ⊗ ǫ F. We are more concerned with the Hilbert tensor product. Its topology is definedby the family { h , i µ ⊗ h , i ν } of non-negative Hermitian forms on E ⊗ F , where h , i µ and h , i ν runs through allnon-negative continuous Hermitian forms on E and F , respectively.Recall that a locally convex space is said to be Hilbertizable if its topology isdefined by a family of non-negative Hermitian forms on it. The following fact is wellknown. Lemma 2.1. If E is nuclear and F is Hilbertizable, then E b ⊗ h F = E b ⊗ F as locally convex spaces.Proof. We indicate the idea of the proof for the convenience of the reader. In general,the Hilbert topology is coarser than the projective topology. If both E and F areHilbertizable, then the epsilon topology is coarser than the Hilbert topology. Thelemma then follows by Grothedieck’s theorem and the well know fact that everynuclear locally convex space is Hilbertizable (cf. [Th, Corallary 3.19]). (cid:3) B. SUN
Smoothing representations.
In this subsection, G is an arbitrary Lie group.Let V be a representation of G (recall from the Introduction that every representationspace is assumed to be complete). It is said to be smooth if the action map G × V → V is smooth as a map of infinite dimensional manifolds. The notion of smooth maps ininfinite dimensional setting may be found in [GN], for example. In general, denoteby C ( G ; V ) the space of continuous functions on G with values in V . It is a completelocally convex space under the usual topology of uniform convergence on compactsets, and is a representation of G under right translations:( g.f )( x ) := f ( xg ) , g, x ∈ G, f ∈ C ( G ; V ) . Similarly, smooth V -valued functions form a complete locally convex space C ∞ ( G ; V )under the usual smooth topology, and is a smooth representation of G under righttranslations.Write γ v ( g ) := g.v, v ∈ V, g ∈ G. The smoothing V ∞ of V is defined to be the representation which fits to a cartesiandiagram V ∞ −−−→ C ∞ ( G ; V ) y y V v γ v −−−→ C ( G ; V )in the category of representations of G . This is a smooth representation of G . As avector space, it consists of all v ∈ V such that γ v ∈ C ∞ ( G ; V ). The topology of V ∞ coincides with the subspace topology of C ∞ ( G ; V ). The universal enveloping algebraU( g C ) acts on V ∞ as continuous linear operators by X.v := the value of X ( γ v ) at the identity element 1 ∈ G. The topology of V ∞ is determined by the family {| · | X,λ } X ∈ U( g C ) , λ ∈ Λ of seminorms, where Λ is the set of all continuous seminorms on V , and | v | X,λ := | X.v | λ . The smoothing is clearly a functor from the category of representations of G tothe category of smooth representations of G . When V itself is smooth, we have that V = V ∞ as representations of G . In general, V ∞ is a dense subspace of V . Lemma 2.2.
Let V be a G -stable subspace of V ∞ . If it is dense in V , then it is alsodense in V ∞ . N REPRESENTATIONS OF REAL JACOBI GROUPS 9
Proof.
Fix a left invariant Haar measure dg on G . Let f ∈ C ∞ ( G ) (a smooth functionwith compact support). For all v ∈ V , write f.v := Z G f ( g ) g.v dg. It is well know and also easy to see that f.v ∈ V ∞ and the linear map V → V ∞ , v f.v is continuous. Then the denseness of V in V implies the denseness of f.V in f.V ,under the topology of V ∞ . Consequently, under the topology of V ∞ ,(11) X f ∈ C ∞ ( G ) f.V is dense in X f ∈ C ∞ ( G ) f.V. Then the lemma follows by noting that the closure of V (within V ∞ ) contains thefirst space of (11), and that the second space of (11) (the Garding subspace) is densein V ∞ . (cid:3) For every closed subgroup S of G , write V | S := V , viewed as a representation of S . Lemma 2.3.
Let S and S ′ be two closed subgroups of G . If the actions of U( s C ) and U( s ′ C ) produce the same subalgebra of End( V ∞ ) , then ( V | S ) ∞ = ( V | S ′ ) ∞ as locally convex spaces.Proof. Apply Lemma 2.2 to the representation V | S , we see that V ∞ is dense in( V | S ) ∞ . Therefore ( V | S ) ∞ is the completion of V ∞ under the seminorms {| · | X,λ } X ∈ U( s C ) , λ ∈ Λ . Similarly, ( V | S ′ ) ∞ is the completion of V ∞ under the seminorms {| · | X,λ } X ∈ U( s ′ C ) , λ ∈ Λ . The assumption of the lemma implies that these two families of seminorms are thesame. Therefore the lemma follows. (cid:3)
We say that a representation is Hibertizable if its underlying space is. The smooth-ing of a Hibertizable representation is also Hibertizable. Let V ′ be another repre-sentation of another Lie group G ′ . Then V b ⊗ π V ′ is a representation of G × G ′ . Ifboth V and V ′ are Hibertizable, then V b ⊗ h V ′ is also a (Hilbertizable) representationof G × G ′ . Lemma 2.4.
If both V and V ′ are smooth, then so is V b ⊗ π V ′ . If both V and V ′ aresmooth and Hibertizable, then so is V b ⊗ h V ′ .Proof. We prove the firs assertion. The second one is proved similarly. The algebraictensor product V ⊗ V ′ is clearly contained in ( V b ⊗ π V ′ ) ∞ . It is dense in V b ⊗ π V ′ and is therefore also dense in ( V b ⊗ π V ′ ) ∞ , by Lemma 2.2. Note that U( g C × g ′ C ) actscontinuously on V ⊗ π V ′ . This implies that the topology of ( V b ⊗ π V ′ ) ∞ and of V b ⊗ π V ′ have the same restriction to V ⊗ V ′ . Consequently, ( V b ⊗ π V ′ ) ∞ and V b ⊗ π V ′ are bothcompletions of V ⊗ V ′ with respect to a common locally convex topology. Thereforethey are the same and the first assertion of the lemma follows. (cid:3) Lemma 2.5.
If both V and V ′ are Hibertizable, then (12) ( V b ⊗ h V ′ ) ∞ = V ∞ b ⊗ h V ′∞ . Proof.
The proof is similar to that of Lemma 2.4 and we will not go to the details.The key point is that both sides of (12) contain V ∞ ⊗ V ′∞ as a dense subspace. (cid:3) It is not clear to the author whether the analog of (12) holds for projective tensorproducts.2.3.
Representations of moderate growth.
We will use the notation of [SZ1] forfunction spaces, as what follows. Let G be an almost linear Nash group. It is notassumed to be a real Jacobi group in this section. We say that a (complex valued)function f on G has moderate growth if its absolute value is bounded by a positiveNash function φ on G : | f ( x ) | ≤ φ ( x ) for all x ∈ G. A smooth function f ∈ C ∞ ( G ) is said to be tempered if Xf has moderate growthfor all X ∈ U( g C ). Denote by C ξ ( G ) the space of all tempered smooth functions on G .A smooth function f ∈ C ∞ ( G ) is called Schwartz if | f | X,φ := sup x ∈ G φ ( x ) | ( Xf )( x ) | < ∞ for all X ∈ U( g C ), and all positive functions φ on G of moderate growth. Denote by C ς ( G ) the space of Schwartz functions on G . It is a nuclear Fr´echet space under theseminorms {| ·| X,φ } . We define the nuclear Fr´echet space D ς ( G ) of Schwartz densitieson G similarly. Fix a Haar measure dg on G , then the map C ς ( G ) → D ς ( G ) ,f f dg is a topological linear isomorphism. We define a tempered generalized function on G to be a continuous linear functional on D ς ( G ). Denote by C − ξ ( G ) the space of alltempered generalized functions on G , equipped with the strong dual topology. This N REPRESENTATIONS OF REAL JACOBI GROUPS 11 topology coincides with the topology of uniform convergence on compact subsets ofD ς ( G ), due to the fact that every bounded subset of a nuclear locally convex space isrelatively compact. Note that C ξ ( G ) is canonically identified with a dense subspaceof C − ξ ( G ): C ξ ( G ) ֒ → C − ξ ( G ) . A representation V of G is said to be of moderate growth if for every continuousseminorm | · | µ on V , there is a positive function φ on G of moderate growth and acontinuous seminorm | · | ν on V such that | gv | µ ≤ φ ( g ) | v | ν , for all g ∈ G, v ∈ V. It is easy to check that the smoothing of a representation of moderate growth isstill of moderate growth. For every representation V of G of moderate growth, thebilinear map(13) D ς ( G ) × V → V ∞ ,λ = f ( g ) dg, v λ.v := R G f ( g ) g.v dg is well defined and continuous. The space D ς ( G ) is an associative algebra under theconvolution operator ∗ . Under the map (13), both V and V ∞ are D ς ( G )-modules.2.4. Matrix coefficients for distributional vectors.
Let G be an almost linearNash group as in the last subsection. Let U h and V h be two Hilbert space repre-sentations of G of moderate growth which are strongly dual to each other in thefollowing sense: there is given a G -invariant continuous bilinear map h , i : U h × V h → C which induces a topological isomorphism from U h to the strong dual of V h (andhence a topological isomorphism from V h to the strong dual of U h ). Denote by U −∞ the strong dual of the smoothing V h ∞ of V h . This is a locally convex space carryinga linear action of G . There are canonical G -intertwining continuous injections U h ∞ → U h → U −∞ . It is not clear to the author whether U h is dense in U −∞ in general. Similarly, denoteby V −∞ the strong dual of the smoothing U h ∞ .Define a bilinear map(14) D ς ( G ) × U −∞ → U h = Hom C ( V h , C ) ,λ = f ( g ) dg, φ λ.φ := (cid:0) v φ (cid:0)R G f ( g ) g − .v dg (cid:1)(cid:1) . The following lemma is elementary and may be proved by the argument of [SZ1,Section 3].
Lemma 2.6.
The map (14) is well defined and its image is contained in U h ∞ . Theresulting bilinear map D ς ( G ) × U −∞ → U ∞ , ( λ, φ ) λ.φ is separately continuous and extends the continuous bilinear map (13) for the repre-sentation U h . Finally, we define the matrix coefficient map c : U −∞ × V −∞ → C − ξ ( G ) = Hom C (D ς ( G ) , C ) ,φ, φ ′ c φ ⊗ φ ′ := ( λ φ ′ ( λ.φ )) . This extends the ordinary matrix coefficient map (see (8)), and is separately contin-uous (see the proof of [SZ1, Lemma 3.6]).3.
Representations of Heisenberg groups
Let w be a finite dimensional real vector space with a non-degenerate skew-symmetric bilinear map h , i w : w × w → z , where dim R (z) = 1.Let H(w) = w × z be the associated Heisenberg group, with group multiplication( u, t )( u ′ , t ′ ) := ( u + u ′ , t + t ′ + h u, u ′ i w ) . Let ψ z be a non-trivial unitary character on z. By Stone-Von Neumann Theorem, upto isomorphism, there is a unique irreducible unitary ψ z -representation ω h ψ z of H(w).Write ω ψ z for its smoothing. It is well known that this is a nuclear Fr´echet space.3.1. Smooth representations.
Recall that J mod H(w) ,ψ z is the category of smoothFr´echet ψ z -representations of H(w) of moderate growth. The following fact is funda-mental to this paper. Proposition 3.1.
For every representation V in J mod H(w) ,ψ z , Hom
H(w) ( ω ψ z , V ) isa Fr´echet space under the topology of uniform convergence on bounded sets. Forevery Fr´echet space E , ω ψ z b ⊗ E is a representation in J mod H(w) ,ψ z . Further more,one has that ω ψ z b ⊗ Hom
H(w) ( ω ψ z , V ) = V, and Hom
H(w) ( ω ψ z , ω ψ z b ⊗ E ) = E. Consequently, the functors
Hom H( W ) ( ω ψ z , · ) and ω ψ z b ⊗ · are mutually inverse equivalences of categories between the category J mod H(w) ,ψ z and the category of Fr´echet space. N REPRESENTATIONS OF REAL JACOBI GROUPS 13
In view of the following lemma, Proposition 3.1 is a combination of [du1, Theorem3.3] and [du3, Theorem 3.4].
Lemma 3.2.
Denote by I ψ z the annihilator of ω ψ z in D ς (H(w)) . Then I ψ z annihi-lates every representation in J mod H(w) ,ψ z . Consequently, every representation in J mod H(w) ,ψ z is a D ς (H(w)) /I ψ z -module.Proof. Fix a Haar measure dw on w and a Haar measure dz on z. Then dh := dw ⊗ dz is a Haar measure on H(w). Define a continuous surjective map(15) D ς (H(w)) → C ς (w) ,λ = f dh ( w R z f ( zw ) ψ ( z ) dz ) . Denote by I ′ ψ z the kernel of this map. It is checked to be a closed ideal of D ς (H( w )).We view C ς (w) as an associative algebra so that the map (15) is an algebra homo-morphism. It is easy to see that all representations in J mod H(w) ,ψ z are annihilatedby I ′ ψ z . Therefore they are all C ς (w)-modules.On the other hand, a classical result of I. Segal says that the action of C ς (w) on ω ψ z is faithful (c.f. [Ho, page 826]). Therefore I ′ ψ z = I ψ z . This proves the lemma. (cid:3) Unitary representations.
As usual, for every complex vector space E , write¯ E for its complex conjugation. This equals to E as a real vector space, and itscomplex scalar multiplication is obtained by composing the complex conjugationwith the scalar multiplication of E .Denote by h , i ψ z the inner product on the Hilbert space ω h ψ z . It restricts to anonzero H(w)-invariant continuous bilinear form h , i ψ z : ω ψ z × ¯ ω ψ z → C . It is well known that such a form is unique up to scalar (see [du4, Proposition 4.12]for example). This implies the following
Lemma 3.3.
For all Fr´echet spaces E and F , the map B( E, F ) → B H(w) ( ω ψ z b ⊗ E, ¯ ω ψ z b ⊗ F ) ,b
7→ h , i ψ z ⊗ b is a linear isomorphism, where “ B ” stands for the space of continuous bilinear forms,and “ B H(w) ” stands for the space of
H(w) -invariant continuous bilinear forms.
Denote by H mod H(w) ,ψ z the category of unitary ψ z -representations of H(w). Forevery Hilbert space E with inner product h , i E , ω h ψ z b ⊗ h E is a unitary representationin H mod H(w) ,ψ z . (The inner product is h , i ψ z ⊗ h , i E .) Conversely, we have Lemma 3.4.
Every representation V in H mod H(w) ,ψ z is unitarily isomorphic to ω h ψ z b ⊗ h E for some Hilbert space E . Proof.
This is well known. We provide a proof for completeness. By Proposition3.1, the smoothing V ∞ has the form ω ψ z b ⊗ E , where E is a certain Fr´echet space.By Lemma 3.3, the restriction of the inner product on V to ω ψ z b ⊗ E has the form h , i ψ z ⊗ b , where b is an inner product on E . Denote by E the completion of E with respect to b . This is a Hilbert space and we have that V = ω h ψ z b ⊗ h E . (cid:3) For every unitary representation V in H mod H(w) ,ψ z , with inner product h , i V ,define an inner product on Hom H(w) ( ω h ψ , V ) by(16) h φ, φ ′ i := φ ′∗ ◦ φ ∈ Hom
H(w) ( ω h ψ z , ω h ψ z ) = C , where φ ′∗ ∈ Hom
H(w) ( V, ω h ψ z ) is determined by the formula h φ ′ ( u ) , v i V = h u, φ ′∗ ( v ) i ψ z , u ∈ ω h ψ z , v ∈ V. The following analog of Proposition 3.1 is an easy consequence of Lemma 3.4.
Proposition 3.5.
For every representation V in H mod H(w) ,ψ z , Hom
H(w) ( ω h ψ z , V ) isa Hilbert space under the inner product (16) . For every Hilbert space E , ω h ψ z b ⊗ h E isa representation in H mod H(w) ,ψ z . Further more, one has inner product preservingidentifications ω h ψ z b ⊗ h Hom
H(w) ( ω h ψ z , V ) = V, and Hom
H(w) ( ω h ψ z , ω h ψ z b ⊗ h E ) = E. Consequently, the functors
Hom H( W ) ( ω h ψ z , · ) and ω h ψ z b ⊗ h · are mutually inverse equivalences of categories between the category H mod H(w) ,ψ z and the category of Hilbert spaces. Representations of Jacobi groups
Let G be a real Jacobi group, and we return to the notation of the Introduction.This section is devoted to a proof of Theorem A. Without lose of generality, weassume in this section that ψ has a discrete kernel. Then Z is either trivial or onedimensional. If it is trivial, then H is trivial and Theorem A is also trivial. So furtherassume in this section that Z is one dimensional. Then H is a Heisenberg group asin last section. N REPRESENTATIONS OF REAL JACOBI GROUPS 15
Splitting Jacobi groups.
Fix a subspace w of the Lie algebra h which iscomplementary to z . Such a space is unique up to conjugations by H . It determinesa Nash splitting i L : L → G of the quotient map G → L so that i L ( L ) stabilizes wunder the adjoint action (cf. [Mo, Theorem 7.1]). Identify L with i L ( L ), then wehave G = L ⋉ H. As in the Introduction, w ∼ = h / z is a symplectic space under the Lie bracket[ , ] : w × w → z . We let the symplectic group Sp(w) acts on H as group automorphisms so that itpoint-wise fixes Z and induces the natural action on w. Let f Sp(w) acts on H throughthe covering map f Sp(w) → Sp(w). Then the following semidirect product is a realJacobi group: e J(w) := f Sp(w) ⋉ H. The adjoint action induces a homomorphism L → Sp(w). Recall from the Introduc-tion that e L := L × Sp(w) f Sp(w)is a double cover of L . The double cover e G := G × L e L = e L ⋉ H of G is obviously mapped to both e J(w) and e L . In this way, we view it as a subgroupof the product e J(w) × e L .4.2. The oscillator representation.
To distinguish representations of differentgroups, we write π | S to emphasize that π is viewed as a representation of a group S . Denote by ω h ψ | e J(w) the unitary oscillator representation of e J(w) corresponding to ψ . Up to isomorphism, this is the only genuine unitary ψ -representation of it whichremains irreducible when restricted to H . Without lose of generality, assume thatthe representation ω h ψ of e G in the introduction coincides with the pull back of ω h ψ | e J(w) to e G .The following lemma is well know (see [Ad, Section 2], for example). Lemma 4.1.
The universal enveloping algebras U( sp (w) C ⋉ h C ) and U( h C ) producethe same subalgebra of End(( ω h ψ | e J(w) ) ∞ ) . Lemma 2.3 then implies that ω ψ := ( ω h ψ | e G ) ∞ = ( ω h ψ | e J(w) ) ∞ = ( ω h ψ | H ) ∞ as Fr´echet spaces. Equivalences of categories.
Recall from the Introduction the categories H mod G,ψ , J mod G,ψ , H mod e L, gen , and J mod e L, gen . Given any representation V in H mod G,ψ , recall from the last section that Hom H ( ω h ψ , V )is a Hilbert space. Let e G act on it by(˜ g.φ )( x ) := g ( φ (˜ g − x )) , ˜ g ∈ e G, φ ∈ Hom H ( ω h ψ , V ) , x ∈ ω h ψ , where g is the image of ˜ g under the quotient map e G → G . This action is checked tobe continuous (use Proposition 3.5) and descends to a genuine representation of e L .Therefore we get a functorHom H ( ω h ψ , · ) : H mod G,ψ → H mod e L, gen . On the other hand, given any representation E in H mod e L, gen , the tensor product ω h ψ b ⊗ h E is a Hilbert space representation of e J(w) × e L . Its restriction to e G descendsto a representation of G . In this way, we get a functor ω h ψ b ⊗ h · : H mod e L, gen → H mod G,ψ . Similarly, we have functors(17) Hom H ( ω ψ , · ) : J mod G,ψ → J mod e L, gen and(18) ω ψ b ⊗ · : J mod e L, gen → J mod G,ψ . Proposition 4.2.
The functors
Hom H ( ω h ψ , · ) and ω h ψ b ⊗ h · are mutually inverse equivalences of categories between H mod G,ψ and H mod e L, gen .Similarly, the functors Hom H ( ω ψ , · ) and ω ψ b ⊗ · are mutually inverse equivalences of categories between J mod G,ψ and J mod e L, gen .Proof. In view of Proposition 3.5, to prove the first assertion, it suffices to show thatthe identification Hom H( W ) ( ω h ψ , ω h ψ b ⊗ h E ) = E respects the e L -actions, and the identification ω h ψ b ⊗ h Hom H( W ) ( ω h ψ , V ) = V respects the G -actions. This is routine to check and we will not go to the details.The second assertion is proved similarly. (cid:3) Corollary 4.3.
Every representation in H mod G,ψ is of moderate growth.
N REPRESENTATIONS OF REAL JACOBI GROUPS 17
Proof.
Since every representation in H mod G,ψ has the form ω h ψ b ⊗ h E ( E is a represen-tation in H mod e L, gen ), this follows from the well know fact that every Banach spacerepresentation of a reductive Nash group is of moderate growth (cf. [Wa, Lemma11.5.1]). (cid:3) Smoothing.
Recall that e G is a Lie subgroup of e J(w) × e L , and therefore g C isa Lie subalgebra of ( sp (w) C ⋉ h C ) × l C .Let E be a representation in H mod e L, gen . Write V ◦ := ω h ψ | e J(w) b ⊗ h E, viewed as a representation of e J(w) × e L . The descent to G of its restriction to e G isdenoted by V . Lemma 2.5 and Lemma 2.1 implies that V ◦∞ = ( ω h ψ | e J(w) ) ∞ b ⊗ h E ∞ = ω ψ | e J(w) b ⊗ h E ∞ = ω ψ | e J(w) b ⊗ E ∞ . Lemma 4.4.
The universal enveloping algebras
U(( sp (w) C ⋉ h C ) × l C ) and U( g C ) produce the same subalgebra of End( V ◦∞ ) .Proof. Write ρ : U(( sp (w) C ⋉ h C ) × l C ) → End( V ◦∞ )for the Lie algebra action. Lemma 4.1 implies that ρ ( sp (w) C ) ⊂ ρ (U( h C )) ⊂ ρ (U( g C )) . The lemma then follows by noting that( sp (w) C ⋉ h C ) × l C = sp (w) C + g C . (cid:3) Finally, by Lemma 2.3, we conclude that V ∞ = V ◦∞ = ω ψ b ⊗ E ∞ . Together with Proposition 4.2, this finishes the proof of Theorem A.5.
Casselman-Wallach ψ -representations and generalized matrixcoefficients We continue to use the notation of the Introduction and assume that G is a realJacobi group. Casselman-Wallach ψ -representations. Recall from the Introduction that
F H
G,ψ is the full subcategory of J mod G,ψ corresponding (under the functors (6)and (7)) to the subcategory
F H e L, gen of J mod e L, gen . Objects in F H
G,ψ are calledCasselman-Wallach ψ -representations of G .The following lemma is well known and is an easy consequenc of Casselman-Wallach’s Theorem ([Wa, Corollary 11.6.8]). Lemma 5.1.
Every injective homomorphism φ : E → F in the category F H e L, gen isa topological embedding with closed image. We use the above lemma to show the following
Lemma 5.2.
Let U be a smooth Fr´echet representation of G of moderate growth, andlet V be a Casselman-Wallach ψ -representation of G . Then every G -intertwiningcontinuous linear map φ from U to V is a topological homomorphism with closedimage.Proof. The map φ descends to an injective homomorphism φ ′ : U/ Ker( φ ) → V in the category J mod G,ψ . By Proposition 4.2, this corresponds to an injectivehomomorphism φ ′′ : E → F in the category J mod e L, gen . Since F is in F H e L, gen , by applying Harish-Chandra’sfunctor to φ ′′ , we see that E is also in F H e L, gen . Then Lemma 5.1 says that φ ′′ isa topological embedding with closed image. This implies that so it φ ′ , since ω ψ b ⊗ · is a topologically exact functor on the category of Fr´echet spaces (cf. [Ta, Theorem5.24]). (cid:3) Now let U = ω ψ b ⊗ E be a representation in F H
G,ψ , and let V = ¯ ω ψ b ⊗ F be arepresentation in F H G, ¯ ψ . Here both E and F are representations in F H e L, gen . Wesay that they are contragredient to each other if there is given a non-degenerate G -invariant continuous bilinear form h , i : U × V → C . Proposition 5.3.
Every representation in
F H
G,ψ has a unique contragredient rep-resentation in
F H G, ¯ ψ .Proof. Lemma 3.3 implies thatB G ( U, V ) = B e L ( E, F ) . Therefore, in view of Proposition 4.2, the Proposition follows from the correspondingresult in the category
F H e L, gen . But the later is a consequence of Casselman-Wallach’sTheorem. (cid:3) N REPRESENTATIONS OF REAL JACOBI GROUPS 19
Generalized matrix coefficients.
Let U , V , E , F be as before. Assume that U and V , and hence E and F , are contragredient to each other. It is well knownthat there is a representation E h in H mod e L, gen so that its smoothing coincides E (cf. [BK, Section 3.1]). Denote by F h the space of continuous linear functionals on E h . As usual, it is a representation in H mod e L, gen which is contragredient to E h .Put U h := ω h ψ b ⊗ h E h and V h := ¯ ω h ψ b ⊗ h F h . View them as representations in H mod G,ψ and H mod G, ¯ ψ , respectively. Theorem Aimplies that U h ∞ = U and V h ∞ = V. Denote by U −∞ the strong dual of V , and by V −∞ the strong dual of U . They areboth representations of G (due to the fact that both U and V are Nuclear Fr´echet,and hence reflexive spaces). Furthermore, U −∞ contains U as a dense subspace, and V −∞ contains V as a dense subspace.Note that U h and V h are strongly dual to each other. By the argument of Section2.4, we get a separately continuous bilinear map(19) U −∞ × V −∞ → C − ξ ( G )which extends the usual matrix coefficient map. Note that all the three spaces in (19)are strong duals of reflexive Fr´echet spaces. Therefore [Tr, Theorem 41.1] impliesthat the map (19) is automatically continuous. Then (19) induces a continuous linearmap(20) c : U −∞ b ⊗ V −∞ → C − ξ ( G ) . By [Tr, Proposition 50.7], U −∞ b ⊗ V −∞ equals to the strong dual of V b ⊗ U . Therefore(20) is the transpose of a G × G -intertwining continuous linear map(21) D ς ( G ) → V b ⊗ U. Apply Lemma 5.2 to the group G × G , we see that the map (21) is a topologicalhomomorphism with closed image. In view of the following lemma, we conclude that(20) is also a topological homomorphism with closed image. This proves CorollaryB. Lemma 5.4. (See [Bo, Section IV.2, Theorem 1] and [SZ1, Lemma 3.8] ) Let φ : E → E be a continuous linear map of nuclear Fr´echet spaces. Denote by E ′ and E ′ the strong duals of E and E , respectively. Then φ is a topological homomorphismif and only if its transpose φ t : E ′ → E ′ is. When this is the case, both φ and φ t have closed images. References [Ad] J. Adams,
The theta correspondence over R , in Harmonic analysis, group representations,automorphic forms and invariant theory , 1-39, Lect. Notes Ser. Inst. Math. Sci. Natl. Univ.Singap., 12, World Sci. Publ., Hackensack, NJ, 2007.[Be] R. Berndt,
On local Whittaker models for the Jacobi group of degree one , Manuscripta Math.84 (1994), no. 2, 177-191.[BK] J. Bernstein and B. Krotz,
Smooth Frechet globalizations of Harish-Chandra modules
Topological vector spaces, Chapter 1-5 , Springer-Verlag, 1987.[Ca] W. Casselman,
Canonical extensions of Harish-Chandra modules to representations of G ,Canad. Jour. Math. 41 (1989), 385–438.[du1] F. du Cloux, Jets de fonctions diff´erentiables sur le dual d’un groupe de Lie nilpotent , Invent.Math. 88 (1987), no. 2, 375-394.[du2] F. du Cloux,
Finite-length representations of nilpotent Lie groups , Symposia Mathematica,Vol. XXXI (1988), 1-29.[du3] F. du Cloux,
Repr´esentations temp´er´ees des groupes de Lie nilpotents , J. Funct. Anal. 85(1989), no. 2, 420-457.[du4] F. du Cloux,
Sur les reprsentations diffrentiables des groupes de Lie algbriques , Ann. Sci.Ecole Norm. Sup. 24 (1991), no. 3, 257-318.[EZ] M. Eichler and D. Zagier,
The theory of Jacobi forms , Progress in Mathematics, 55, BirkhauserBoston, 1985.[Ho] R. Howe,
On the role of the Heisenberg group in harmonic analysis , Bull. Amer. Math. Soc.(N.S.) 3 (1980), no. 2, 821-843.[GN] H. Gl¨ockner and K.-H. Neeb,
Infinite-dimensional Lie groups: General Theory and MainExamples , Graduate Texts in Mathematics, Springer-Verlag, book to appear.[Mo] G. D. Mostow,
Fully reducible subgroups of algebraic groups , Amer. J. Math. 78 (1956), 200-221.[Sh1] M. Shiota,
Nash manifolds , Lecture Notes in Math. 1269, Springer-Verlag, 1987.[Sh2] M. Shiota,
Nash functions and manifolds , in F. Broglia (ed.),
Lectures in Real Geometry . deGruyter expositions in Mathematics, vol. 23 (1996).[SZ1] B. Sun and C.-B. Zhu,
A general form of Gelfand-Kazhdan criterion , arXiv:0903.1409.[SZ2] B. Sun and C.-B. Zhu,
Multiplicity one theorems: the archimedean case , preprint.[Ta] J.L. Taylor,
Notes on locally convex topological vector spaces ∼ taylor/LCS.pdf, 1995.[Th] E. Thomas, Nuclear spaces and topological tensor products ∼ thomas/.[Tr] F. Treves, Topological vector spaces, distributions and kernels , Academic Press, New York,1967.[Wa] N. Wallach,
Real Reductive Groups II , Academic Press, San Diego, 1992.
Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Bei-jing, 100190, P.R. China
E-mail address ::