A Laplace-type representation of the generalized spherical functions associated to the root systems of type A
aa r X i v : . [ m a t h . R T ] A ug A Laplace-type representation of the generalizedspherical functions associated to the root systems oftype A Patrice Sawyer
Department of Mathematics & Computer Science, Laurentian University, Sudbury,Ontario, Canada K1K 4W5
Abstract
In this paper, we extend the iterative expression for the generalized sphericalfunctions associated to the root systems of type A previously obtained beyondregular elements. We also provide the corresponding expression in the flatcase. From there, we derive a Laplace-type representation for the generalizedspherical functions associated to the root systems of type A in the Dunklsetting as well as in the trigonometric Dunkl setting. This representationleads us to describe precisely the support of the generalized Abel transform.Thanks to a recent result of Rejeb, this allows us to give the support for theDunkl intertwining operator. Keywords: generalized spherical function, Dunkl, root system, intertwiningoperator, Abel transform, dual of the Abel transformThis research is supported by funding from Laurentian University.The author is thankful to the Institut f¨ur Mathematik at the Universi¨atPaderborn for their hospitality in July 2013 during which this work wasstarted and to Professor Margit R¨osler for helpful conversations.
1. Introduction
We start by providing some background. We refer the reader to [10, 11,15] for a more complete exposition on the Dunkl and trigonometric Dunklsettings. Given a root system R and a Cartan subalgebra a , for every root α ∈ R , let r α ( X ) = X − h α,X ih α,α i α and let ∂ ξ be the derivative in the direction Preprint submitted to Elsevier October 6, 2018 f ξ . The Dunkl operators indexed by ξ ∈ a are then given by T ξ = ∂ ξ + X α ∈ R + k α α ( ξ ) 1 h α, X i (1 − r α )The Weyl group W associated to the root system is generated by the reflec-tion maps r α .In a similar manner, the trigonometric Dunkl operators (also called Dunkl-Cherednik operators or simply Cherednik operators) are given by D ξ = ∂ ξ + X α ∈ R + k α α ( ξ ) 11 − e − α (1 − r α ) − ρ ( k )( ξ ) . In the Dunkl setting, the function G ( λ, · ) is defined as the unique analyticsolution of T ξ G ( λ, · ) = h ξ, λ i G ( λ, · ) (1)with G ( λ,
0) = 1.We can also define the generalized spherical functions as follows: F ( λ, · )is the unique analytic function such that for every symmetric polynomial p ( i.e. a polynomial which is invariant with respect to the action of the Weylgroup), we have p ( T ξ ) F ( λ, · ) = p ( h ξ, λ i ) F ( λ, · ) (2)with F ( λ,
0) = 1. Note that F ( λ, X ) = 1 | W | X w ∈ W G ( λ, w · X ) . (3)The definitions in (1), (2) and (3) are essentially the same in the trigono-metric setting except that we then replace T ξ by D ξ (refer for example to[10, 11]).With some adjustment in the spectral parameter λ , the functions F ( λ, · )generalize the spherical functions on symmetric spaces of Euclidean type (inthe Dunkl setting) and those on the symmetric spaces of noncompact type(in the trigonometric Dunkl setting). We refer the reader to Helgason’s books[7, 8] as the standard reference on symmetric spaces.2ndeed, for selected choice of root multiplicities, the root system corre-sponds to a symmetric space of noncompact type in the trigonometric Dunklsetting and to the corresponding flat symmetric space in the Dunkl setting.We then say that we are in the “group case” or “in the geometric setting”.For example, in the case of the trigonometric setting for the root system oftype A n − , m = 1, 2 or 4 corresponds to the spaces SL ( n, F ) / SU ( n, F ) with F = R , C or H and m = dim R F (in addition, when n = 3, m = 8 also givethe space SL (3 , O ) / SU (3 , O ) ≃ E − / F where O denotes the octonions).In the Dunkl setting, for the same choice of the multiplicity m , we have thecorresponding flat symmetric spaces.It is well-know that in the geometric setting, if H = 0 then the sphericalfunctions have a Laplace type representation φ λ ( e X ) = Z a e i h λ,H i K ( H, X ) dH (4)with K ( H, X ) > K ( · , X ) is C ( X ) ( H = 0 ensuresthat dim C ( X ) = rank by [8, Theorem 10.1, Chap. IV]).The function K ( H, · ) is the kernel of the Abel transform A ( f )( H ) = Z a f ( e X ) K ( H, X ) δ ( X ) dX while the dual Abel transform is simply given by A ∗ ( f )( X ) = Z a f ( e H ) K ( H, X ) dH so that φ λ = A ∗ ( e i h λ, ·i ).As for the function G ( λ, · ) in the Dunkl setting, we have the followingrepresentation G ( λ, · ) = V e h λ, ·i where V is called the Dunkl’s intertwining operator. It is defined by thefollowing properties T ξ V = V ∂ ξ ,V ( P n ) = P n ,V | P = id3here P n is the space of homogeneous polynomials of degree n . We alsointroduce the positive measure µ x such that V f ( X ) = Z a f ( H ) dµ X ( H )(for the existence of the positive measure, see for example [16]).Compare with the intertwining properties of the generalized Abel trans-form and its dual p ( ∂ ξ ) ◦ A = A ◦ p ( T ξ ) ,p ( T ξ ) ◦ A ∗ = A ∗ ◦ p ( ∂ ξ )for every symmetric polynomial p .From now on, unless otherwise mentioned, we are only concerned withthe root systems of type A . The superscript ( m ) on the various objects willserve to indicate that the associated multiplicity is equal to m ( e.g. φ ( m ) λ , K ( m ) ( H, X ), etc.).In Section 2, we recall the recursive formulae (equation (5) and (6)) for thegeneralized spherical functions φ ( m ) λ (denoted F ( λ, · ) above) associated withthe root system A n − with root multiplicity ℜ m > X ∈ a + (see for instance [17, 18]).We show first that (6) makes sense for all X ∈ a . We then derive Theorem2.2 and Theorem 2.3 to extend (5) and (6) to the cases where X ∈ a is notregular. These two results are interesting in themselves.In Section 3, we show that in the case of the root system of type A n − with arbitrary multiplicity ℜ m >
0, equation (4) still holds with the kernel K ( m ) ( · , X ) supported in the set C ( X ) and that when m > K ( m ) ( · , X ) > C ( X ) ◦ .In Section 4, with the help of a theorem of de Jeu [1], we extend theresults of Section 3 to the Dunkl setting. We also use a result by Rejeb todeduce the exact support of the intertwining transform V .
2. The generalized spherical function associated to the root sys-tems of type A We recall some preliminary definitions and results from [17, 18]. In par-ticular, we describe here the family of differential operators which are in-4trumental in defining the generalized spherical functions related to the rootsystem A n − .In what follows, a is the space of real n × n diagonal matrices and a + is the subset with strictly decreasing diagonal entries. For simplicity, wewill not assume here that the matrices have trace equal to 0 (refer howeverto Remark 2.1). We will use lowercase to write the diagonal entries of anelement of a (e.g. if X ∈ a then X = diag[ x , . . . , x n ]). We describe theaction of the Weyl group on the elements of a as follows: if σ ∈ W = S n then σ · X = diag[ x σ − (1) , . . . , x σ − ( n ) ].The differential operators D ( m )1 , . . . , D ( m ) n defined below generate thealgebra of differential operators p ( D ξ ), where p is any symmetric polynomial. Definition 2.1.
Let Y be an indeterminate, δ = ( n − , n − , . . . , , andlet D n ( Y ; m ) = Y p 1) ( n − 2) (3 n − . Definition 2.2. The generalized spherical function φ ( m ) λ for the root system n − is the unique analytic solution of the system D n ( Y ; m ) φ ( m ) λ ( e X ) = n Y k =1 ( Y + ( n − / i λ k /m ) φ ( m ) λ ( e X ) with φ ( m ) λ ( e ) = 1 . In [17, 18], we proved the following result for the generalized sphericalfunctions associated to the root system A n − . Theorem 2.1. For X ∈ a + , we define φ ( m ) λ ( e X ) = e i λ ( X ) when n = 1 and,for n ≥ , φ ( m ) λ ( e X ) = Γ( m n/ m/ n e i λ n P nk =1 x k Z E ( X ) φ ( m ) λ ( e ξ ) S ( m ) ( ξ, X ) d ( ξ ) m dξ (5) where E ( X ) = { ξ = ( ξ , . . . , ξ n − ) : x k +1 ≤ ξ k ≤ x k } , λ ( X ) = P nj =1 λ j x j , λ ( ξ ) = P n − i =1 ( λ i − λ n ) ξ i , d ( X ) = Q r For X ∈ a , we will write π ( X ) for the unique element in ( W · X ) ∩ a + (the projection of X into a + ). We will also write π ( X ) = diag [ a , . . . , a , a , . . . , a , . . . , a r , . . . , a r ] (8) where π ( X ) = σ · X ∈ a + for some σ ∈ W and where the a i ’s are distinct,decreasing and the size of a given block of a i ’s is n i . We will also use thenotation N = 0 , N k = n + · · · + n k when ≤ k ≤ r (observe that N r = n ). We then need the following auxiliary result. Lemma 2.1. If X ∈ a + then (7) is equivalent to e ξ i , i = 1 , . . . , n − , being theroots of q ( x ) = n X p =1 β p Y i = p ( x − e x i ) , x an indeterminate (9)i.e. q ( x ) = Q n − i =1 ( x − e ξ i ) . Let X ∈ a . If β ∈ σ n then the roots u ≥ u ≥ · · · ≥ u n − of q ( x ) in (9) are strictly positive and if we write ξ i = (log u i ) / , i = 1 , . . . , n − , then ξ ∈ E ( π ( X )) . The map χ λ can be extended continuously to a Weyl-invariant map over a by using equation (6) where u i = e ξ i , i = 1 , . . . , n − , are the rootsof q ( x ) in (9).Proof. 1. Write e k ( µ ) = P i < ··· } 6 = ∅ since P rk =1 γ k = 1. We have q ( x ) = r Y j =1 ( x − e a j ) n j − r X k =1 γ k Y j = k ( x − e a j )= r Y j =1 ( x − e a j ) n j − Y s S ( x − e a s ) X s ∈ S γ s q ( x ) z }| {Y j = s,j ∈ S ( x − e a j ) . For s ∈ S , q ( e a s ) alternate signs: there are | S | − x = e a j with multiplicities n j − x = e a j with j S .3. It suffices to reflect that in expression (6), the variable ξ depends con-tinuously on the coefficients β p (up to their order). Using induction,the rest follows.We are looking for simplified expressions corresponding to (5) and (6)which are also valid for X ∈ a + . The following lemma is the basis for thecomputations required to prove Theorem 2.2. Lemma 2.2. We have Z σ n f ( β + · · · + β k , β k +1 , . . . , β n ) ( β · · · β n ) m/ − dβ = Γ( m/ k Γ( m k/ Z σ n +1 − k f ( γ k , . . . , γ n ) γ m k/ − k ( γ k +1 · · · γ n ) m/ − dγ. Proof. It suffices to use the change of variables γ k = β + · · · + β k , ˜ β i = β i /γ k , i = 1, . . . , k and γ j = β j , j = k + 1, . . . , n and to “integrate out” ˜ β , . . . , ˜ β k noting that P ki =1 ˜ β i = 1 and P nj = k γ k = 1.This brings us to the following result which fully extends (6).8 heorem 2.2. Let X and n , . . . , n r be as in (8) and let N k = n + · · · + n k ( N = 0 ). Then χ ( m ) λ ( e X ) = Γ( m n/ Q ri =1 Γ( m n i / e i λ n P rk =1 n k a k Z σ r χ ( m ) λ ( e ξ ) r Y i =1 γ m n i / − i dγ (10) where the e ξ i are the roots of q ( x ) = Q rj =1 ( x − e a j ) n j − P ri =1 γ i Q j = i ( x − e a j ) .More precisely, ξ = diag[ n − z }| { a , . . . , a , η , n − z }| { a , . . . , a , η , . . . , (11) n − z }| { a r − , . . . , a r − , η r − , n r − z }| { a r , . . . , a r ] ∈ E ( X ) with ξ k = a i whenever N i < i < N i +1 and ξ N i = η i ∈ [ a i , a i +1 ] are the rootsof the polynomial q ( x ) = P ri =1 γ i Q j = i ( x − e a j ) . Equivalently, γ p = Q r − j =1 ( e η j − e a p ) Q j = p ( e a j − e a p ) , p = 1 , . . . , r. Proof. This follows from equation (6) and repeated use of Lemma 2.2. Theorem 2.3. We use the notation of Theorem 2.2 and assume that X ∈ a + for simplicity. Then φ ( m ) λ ( e X ) = Γ( m n/ Q ri =1 Γ( m n i / e i λ n P rk =1 n k a k Z E ( X ) φ ( m ) λ ( e ξ ) ˜ S ( m ) ( η, X ) (12) · Y i We have γ p = Q r − i =1 ( e ηi − e ap ) Q i = p ( e ai − e ap ) , p = 1, . . . , r . Now, ∂ ( γ , . . . , γ r − ) ∂ ( η , . . . , η r − ) = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) det " e η q Q i = q ( e η i − e a p ) Q i = p ( e a i − e a p ) ≤ p,q ≤ r − (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) = 2 r − Q r − p =1 Q r − i =1 ( e η i − e a p ) Q r − p =1 Q i = p ( e a i − e a p ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) det (cid:20) − e a p e − η q (cid:21) ≤ p,q ≤ r − (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) = 2 r − Q i 10e have used (14), (15), Remark 2.1 and the fact that P n − k =1 ξ k = P rk =1 ( n k − a k + P r − k =1 η k . 3. Representation of the spherical function We will now derive a Laplace type representation for the spherical func-tions namely φ ( m ) λ ( e X ) = Z a e i h λ,H i K ( m ) n ( H, X ) dH and, at the same time, specify the support of the function K ( m ) n ( · , X ).Recall that C ( X ), the support of the spherical functions in the geometriccase, is the convex envelop of W · X , where W is the Weyl group. In thissection, we will show that C ( X ) remains the support in the case of thegeneralized spherical functions (specifically when m > φ ( m ) λ , it is not surprising that our first step is to derive an inductivedescription of C ( X ). Definition 3.1. Fix X = diag[ x , . . . , x n ] ∈ a + . For H ∈ C ( X ) , let H ′ =diag[ h , . . . , h n − ] . For Y ∈ C n − ( ξ ) , ξ ∈ E ( X ) , let ˆ Y = diag[ y , . . . , y n − , P nk =1 x k − P n − k =1 ξ k ] and [ C ( ξ ) = { ˆ Y : Y ∈ C ( ξ ) } . The following characterization of C ( X ) and C ( X ) ◦ will prove useful. Remark 3.1 ([12]) . If X ∈ a + then H ∈ a belongs to C ( X ) if and only if tr X = tr H and h i + · · · + h i k ≤ x + · · · + x k (16) for every choice of distinct indices i , . . . , i k ∈ { , . . . , n } , ≤ k ≤ n − .Moreover, H ∈ C ( X ) ◦ if and only if all the inequalities in (16) are strict. Proposition 3.1. If X ∈ a + then C n ( X ) = ∪ ξ ∈ E ( X ) \ C n − ( ξ ) .Proof. We first show that A ( X ) = ∪ ξ ∈ E ( X ) \ C n − ( ξ ) is a convex set. Let H and ˜ H ∈ A ( X ). We have H ′ = P σ ∈ S n − σ · ξ and ˜ H ′ = P σ ∈ S n − σ · ˜ ξ . Forfor t ∈ [0 , t H ′ + (1 − t ) ˜ H ′ = t X σ ∈ S n − σ · ξ + (1 − t ) X σ ∈ S n − σ · ˜ ξ = X σ ∈ S n − σ · ( t ξ + (1 − t ) ˜ ξ ) . E ( X ) is convex, we have η = t ξ + (1 − t ) ˜ ξ ∈ E ( X ). Finally, it followseasily that t H + (1 − t ) ˜ H ∈ [ C ( η ).We next show that C ( X ) ⊂ A ( X ). Given that A ( X ) is convex, weonly need to show that σ · X ∈ A ( X ) for every σ ∈ S n . For such a σ ,we have σ · X = diag[ x σ − , . . . , x σ − ( n ) ]. Now define τ ∈ S n − by τ − ( k ) = (cid:26) σ − ( k ) if σ − ( k ) < σ − ( n ) σ − ( k ) − k ≥ σ − ( n ) for k = 1, . . . , n − ξ by ξ k = (cid:26) x k if k < σ − ( n ) x k +1 if σ − ( k ) ≥ σ − ( n ) for k = 1, . . . , n − 1. One verifies that ξ ∈ E ( H )and that σ · X = d τ · ξ ∈ \ C n − ( ξ ) ⊂ A ( X ).We now claim that A ( X ) ⊂ C ( X ). Let H ∈ [ C ( ξ ) for some ξ ∈ E ( X ).Since H = P τ ∈ S n − b τ d τ · ξ , it suffices to observe that d τ · ξ = ˜ τ · ˆ ξ where˜ τ ∈ S n is defined by ˜ τ ( k ) = τ ( k ) if k < n and ˜ τ ( n ) = n . Since ˆ ξ ∈ C ( X )(which can be checked using Remark 3.1), the claim follows.The next result sets the stage for the proof that the kernel H K ( H, X )is strictly positive on C ( X ) ◦ . Proposition 3.2. Suppose X ∈ a + , X = C I n , and use the notation inTheorem 2.2. Let E ( X ) ◦ = { ξ ∈ E ( X ) : a k +1 < η k < a k } where η k = ξ N k , ≤ k ≤ r − . Then C ( X ) ◦ = ∪ ξ ∈ E ( X ) ◦ \ C n − ( ξ ) ◦ (if n = 2 then assume C n − ( ξ ) = C n − ( ξ ) ◦ = { ξ } ).Proof. Let H ∈ C ( X ) ◦ . We know by Proposition 3.1 that there exists ξ ∈ E ( X ) with H ′ ∈ C ( ξ ). We prove first that ξ can be chosen so that H ′ ∈ C ( ξ ) ◦ . From Remark 3.1, for every distinct i , . . . , i n − ∈ { , . . . , n − } : h + · · · + h n − = ξ + · · · + ξ n − and h i + · · · + h i k ≤ ξ + · · · + ξ k , ≤ k < n − . (17)Suppose H ′ C ( ξ ) ◦ and assume h i ≤ · · · ≤ h i n − . Let k < n − ξ i < x i for some i ≤ k (otherwise h i + · · · + h i k = ξ + · · · + ξ k = x + · · · + x k which contradicts H ∈ C ( X ) ◦ ). We must also have ξ j > x j +1 for some j > k ;otherwise h i + · · · + h i k = ξ + · · · + ξ k and h i + · · · + h i n − = ξ + · · · + ξ n − would mean h i k +1 + · · · + h i n − = x k +2 + · · · + x n i.e. P r = is,s ≥ k +1 h r = x + · · · + x k +1 which contradicts H ∈ C ( X ) ◦ . Let ˜ ξ i = ξ i + δ , ˜ ξ j = ξ j − δ and ˜ ξ s = ξ s if s = i and s = j where 0 < δ < min { x i − ξ i , ξ j − x j +1 } . We replace ξ by ˜ ξ and note12hat H ′ ∈ C ( ˜ ξ ). If H ′ C ( ˜ ξ ) ◦ then the smallest k for which an inequality in(17) is not strict will be larger. Eventually, the process will stop.We may now assume that H ′ ∈ C ( ξ ) ◦ . We now show that we can select ξ with η i < a i for i = 1, . . . , r − 1. Suppose ξ does not satisfy that condition.Let k ≤ r − η k = a k . If k = 1, observe thatthere will be an index j > η j < a j (otherwise h i + · · · + h i n − = x + · · · + x n − which contradicts H ∈ C ( X ) ◦ ); let ˜ η = η − δ , ˜ η j = η j + δ and ˜ η i = η i if i = 1 and i = j with 0 < δ < min { η − a = a − a , a j − η j , ξ + · · · + ξ s − h i − · · · − h i s , ≤ s ≤ j } . We replace ξ by ˜ ξ by changing η and ˜ η . We may now assume that k > 1; let ˜ η = η + δ , ˜ η k = η k − δ and˜ η i = η i for i = 1 and i = k with 0 < δ < min { a − η , a k − a k +1 } . We stillhave H ′ ∈ C ( ˜ ξ ) ◦ and the smallest index such that ˜ η i = a i , if any, will belarger. Eventually, the process will stop.We now assume that H ∈ C ( ξ ) ◦ and that ξ satisfies η i < a i for each i .To ensure that η i > a i +1 for all i , we proceed in much the same way exceptthat we consider the largest index such that η k = a k +1 (if any). The rest isas before.Now let H ∈ \ C ( ξ ) ◦ with ξ ∈ E ( X ) ◦ and suppose H C ( X ) ◦ . Since H ′ ∈ C ( ξ ) ◦ , we must have h i + · · · + h i s − + h n = x + · · · + x s for some s < n .Therefore, since H ′ ∈ C ( ξ ) ◦ and ξ ∈ E ( X ) ◦ , ξ + · · · + ξ n − = h + · · · + h n − = h + · · · + h n − + h n − h n = x + · · · + x n − h n = ( h i + · · · + h i s − + h n )+( x s +1 + · · · + x n ) − h n = ( h i + · · · + h i s − )+( x s +1 + · · · + x n ) < ( ξ + · · · + ξ s − )+( ξ s + · · · + ξ n − )which is absurd. Remark 3.2. Suppose H ∈ C ( X ) ◦ and H ′ ∈ C ( ξ ) with ξ ∈ E ( H ) . Bychoosing the successive δ ’s in the proof small enough, one can choose ˜ ξ ∈ E ( X ) ◦ to be arbitrarily close to ξ with H ′ ∈ C ( ˜ ξ ) ◦ . We can now provide a Laplace-type representation for the generalizedspherical function associated to the root system A n − along with the supportof the dual of the Abel transform. Theorem 3.1. Assume m > and suppose φ ( m ) λ is the generalized sphericalfunction for the root system A n − in the trigonometric setting and, for X ∈ a , X = c I n and H ∈ C ( X ) , let D H ( X ) = { ξ ∈ E ( X ) : H ′ ∈ C ( ξ ) } . Then φ ( m ) λ ( e X ) = Z C ( X ) e i λ ( H ) K ( m ) n ( H, X ) dH (18)13 here K ( m ) n ( H, X ) = Γ( m n/ Q ri =1 Γ( m n i / Z D H ( X ) K ( m ) n − ( H ′ , ξ ) ˜ S ( m ) ( η, X ) Y i If we consider the larger range ℜ m > then the result remainsvalid except that K ( m ) n ( · , X ) = 0 on C ( X ) ◦ does not necessarily follow. Allwe can conclude then is that support K ( m ) n ( · , X ) ⊂ C ( X ) .In the geometric case, when X = c I n , the measure in (18) is the Diracmeasure δ X . Using Proposition 10 and induction, we can conclude that thesame holds in the generalized setting.Proof. Assume X ∈ a , X = c I n . Since φ ( m ) λ ( e X ) = φ ( m ) λ ( e π ( X ) ), we canassume without loss of generality that X ∈ a + . We prove the result byinduction on n ≥ 2. If n = 2 then we have K ( m )2 ( H, X ) = Γ( m )(Γ( m/ sinh − m ( x − x ) [sinh( x − h ) sinh( h − x )] m/ − (20)which is smooth and strictly positive when x > h ≥ h > x i.e. when H ∈ C ( X ) ◦ ( X ∈ a + , X = C I implies X ∈ a + ). Note that if m = 2, K ( m )2 ( H, X ) is either equal to 0 (when m > 2) or infinite (when 0 < m < H ∈ ∂C ( X ); K (2)2 ( H, X ) > H ∈ C ( X ) and is 0 elsewhere.Assume now that the result is true for n − n ≥ 3. Suppose first that r > 2. For H ∈ C ( X ) ◦ , let D H ( X ) = { ξ ∈ E ( X ) : H ′ ∈ C ( ξ ) } ,D ◦ H ( X ) = { ξ ∈ E ( X ) ◦ : H ′ ∈ C ( ξ ) ◦ } , (21) E H ( X ) = { ξ ∈ E ( X ) : tr ξ = tr H ′ } . η , . . . , η r − since h + · · · + h n − = ξ + · · · + ξ n − = η + · · · + η r − + P ri =1 ( n i − a i = h + · · · + h n − . Furthermore, D ◦ H ( X ) ⊆ D H ( X ) ⊆ E H ( X ). Both D H ( X ) and E H ( X )are closed sets. We claim that that D ◦ H ( X ) is a nonempty open subset of E H ( X ) which is dense in D H ( X ). Indeed, observe first that D ◦ H ( X ) = ∅ isa consequence of Proposition 3.2. Now, let ˜ ξ ∈ D ◦ H ( X ) and let 0 < ǫ < min { ( ξ + · · · + ξ s − h i − · · · − h i s ) /s, s = 1 , . . . , n − , a i − η i , η i − a i +1 } (the indices i s are assumed to be distinct and between 1 and n − ξ ∈ E H ( X ) and | η k − ˜ η k | < ǫ for all k then ξ ∈ E ( X ) ◦ and H ′ ∈ C ( ξ ) ◦ i.e. that ξ ∈ D ◦ H ( X ). The fact that D ◦ H ( X ) isdense in D H ( X ) follows easily from Remark 3.2. We have φ ( m ) λ ( e X ) = Γ( m n/ Q ri =1 Γ( m n i / e i λ n P rk =1 n k a k (22) · Z E ( X ) Z C ( ξ ) e i λ ( H ′ ) K ( m ) n − ( H ′ , ξ ) dH ′ ˜ S ( m ) ( η, X ) Y i Graczyk and Loeb have given a fairly explicit construction forthe kernel of the Abel transform in the case of complex Lie groups in [2]while Graczyk and Sawyer have also given in [6] an expression in the caseof Lie groups of noncompact type (with a few exceptions of low dimension). n either case, the support of the kernel is not obvious from the expression(although known since we are in the geometric setting). In [19], Trim`echehas provided sensibly the same expression as we have here in the case A ( n = 3 ) and showed that the support of the kernel is included in C ( X ) . 4. The Dunkl setting We use a result of de Jeu in [1] (taking “rational limits”) to adapt theresults of Section 2 and Section 3 to the Dunkl setting. Using a result fromRejeb, this allows us in turn to describe precisely the support of the Dunklintertwining operator V . Theorem 4.1. We use the notation set up in Definition 2.3. The generalizedDunkl spherical function associated to the root system A n − in the Dunklsettingis given by ψ ( m ) λ ( e X ) = e i λ ( X ) when n = 1 and, for n ≥ , ψ ( m ) λ ( e X ) = Γ( m n/ Q ri =1 Γ( m n i / e i λ n P rk =1 n k a k Z E ( X ) ψ ( m ) λ ( e ξ ) T ( m ) ( η, X ) d ( η ) m dη (23) where X ∈ a + , E ( X ) , and λ are as before, d ( η ) = Q r We apply the Weyl-invariant version of [1, Theorem 4.13]: we have ψ ( m ) λ ( e X ) = lim ǫ → φ ( m ) λ/ǫ ( e ǫ X )uniformly in λ and X over compact sets (this technique has been describedas “taking rational limits”). We then use the change of variable η = ǫ ˜ η whichmeans that ξ = ǫ ˜ ξ and dη = ǫ r − d ˜ η . Counting the powers of ǫ carefully, wehave φ ( m ) ǫ λ ( e ξ ) ˜ S ( m ) ( η, ǫ X ) Y i We use the same definitions as in the theorem. Then ψ ( m ) λ ( e X ) = Γ( m n/ Q ri =1 Γ( m n i / e i λ n P rk =1 n k a k Z σ r ψ ( m ) λ ( e ξ ) n Y i =1 γ m n i / − i dγ (25) where the ξ i are the roots of q ( x ) = Q rj =1 ( x − a j ) n j − P ri =1 γ i Q j = i ( x − a j ) .More precisely, ξ is as in (11) with ξ k = a i whenever N i − < k < N i and ξ N i = η i ∈ [ a i , a i +1 ] are the roots of the polynomial q ( x ) = P ri =1 γ i Q j = i ( x − a j ) . Equivalently, γ p = Q r − j =1 ( η j − a p ) Q j = p ( a j − a p ) , p = 1 , . . . , r. Proof. It suffices to use the change of variables γ p = Q r − j =1 ( η j − a p ) Q j = p ( a j − a p ) , p = 1 , . . . , r in (23). Proposition 4.1. Let f V f ( X ) = R a f ( H ) dµ X ( H ) be the intertwiningoperator in the Dunkl setting as discussed in the Introduction and let f ∗ f the dual Abel operator. If f is a Weyl-invariant smooth function then V f = A ∗ f . support ( µ X ) is Weyl-invariant. support ( K ( · , X )) = support ( µ X ) ⊂ C ( X ) Proof. 1. Suppose that f is Weyl-invariant. Then for X ∈ a , A ∗ f ( X ) = A ∗ (cid:18)Z a ∗ ˜ f ( λ ) e − i λ ( X ) dλ (cid:19) = Z a ∗ ˜ f ( λ ) A ∗ ( e − i λ ( X ) ) dλ = Z a ∗ ˜ f ( λ ) ψ − λ ( X ) dλ while V f ( X ) = V (cid:18)Z a ∗ ˜ f ( λ ) e − i λ ( X ) dλ (cid:19) = V Z a ∗ ˜ f ( λ ) 1 | W | X σ ∈ W e − i λ ( σ − · X ) dλ ! = Z a ∗ ˜ f ( λ ) V | W | X σ ∈ W e − i λ ( σ − · X ) ! dλ = Z a ∗ ˜ f ( λ ) ψ − λ ( X ) dλ the last equality being a consequence of the relationship between thegeneralized spherical functions and the eigenfunctions of the Dunkloperators as given in (3) (refer to [11] for example).2. This result appears in the doctoral thesis of C. Rejeb [13, Theorem2.9].3. support( K ( · , X )) ⊂ support( µ X ). Indeed, suppose that H support( µ X ).Using 2., this implies that ( W · H ) ∩ support( µ X ) = ∅ . Let ǫ > W · B ( H , ǫ )) ∩ support( µ X ) = ∅ and let f be a smoothnon-negative function which is identically 1 on B ( H , ǫ ) and 0 outside B ( H , ǫ ). Let f W ( Z ) = | W | P σ ∈ W f ( σ − · Z ). Then for X ∈ a ,0 = V ( f W )( X ) = A ∗ f W ( X ) = A ∗ f ( X )which means that H support( K ( · , X )).On the other hand, if µ X ( U ) > U open, then A ∗ ( U )( X ) = A ∗ (cid:18) P w ∈ W U ◦ w − | W | (cid:19) ( X ) = V (cid:18) P w ∈ W U ◦ w − | W | (cid:19) ( X ) ≥ V ( U )( X ) | W | > µ X ) ⊂ support( K ( · , X )).18 emark 4.1. R¨osler and Voit have shown in [14, 16] that the support of theintertwining operator V is always included in C ( X ) . Furthermore, our resultshows that the support of the Abel transform and of the intertwining operatorare the same. We show below that the results for the Laplace-type representation of thegeneralized spherical functions associated to the root systems of type A stillhold in the Dunkl setting. Furthermore, we are able to describe precisely thesupport of the Dunkl intertwining operator V . Theorem 4.2. Assume m > and suppose ψ ( m ) λ is the generalized sphericalfunction for the root system A n − in the Dunkl setting and, for X ∈ a , X = c I n and H ∈ C ( X ) , let D H ( X ) = { ξ ∈ E ( X ) : H ′ ∈ C ( ξ ) } . Then ψ ( m ) λ ( e X ) = Z C ( X ) e i λ ( H ) ˇ K ( m ) n ( H, X ) dH where ˇ K ( m ) n ( H, X ) = Γ( m n/ Q ri =1 Γ( m n i / Z D H ( X ) ˇ K ( m ) n − ( H ′ , ξ ) T ( m ) ( η, X ) (26) · Y i 2) ˇ K ( m ) n − ( H ′ , ξ ) T ( m ) ( η, X ) . Furthermore, if n = 2 then K ( m ) n − ( H ′ , ξ ) should be replaced by 1. Remark 3.3 also applies here. Proof. The proof is practically identical to the one of Theorem 3.1.Furthermore, we are able to specify precisely the support of the Dunklintertwining operator V . Corollary 4.2. The support of the intertwining operator V is C ( X ) .Proof. A consequence of the theorem and of Proposition 4.1, part 3.19 . Conclusion Our approach is heavily dependent on the iterative formulae of the spheri-cal functions for the roots system A n − . It is not out of question that iterativeformulae such as (5), (6), (23) and (25) could be developed for the classicalreal and complex symmetric spaces. These formulae could then potentiallylead to expressions for the generalized spherical functions associated to thecorresponding root systems. Not only was this approach used to derive theoriginal formulae (5) and (6) but the same principle was used to prove a sharpcriterion for the existence of the product formula for different root systems(see for instance [3, 4, 5]). References [1] M. de Jeu. Paley-Wiener theorems for the Dunkl transform , Trans. Amer.Math. Soc. 358 (2006), no. 10, 4225–4250.[2] P. Graczyk and J.-J. Loeb. Bochner and Schoenberg theorems on sym-metric spaces in the complex case , Bull. Soc. Math. France 122 (1994),no. 4, 571–590.[3] P. Graczyk, P. Sawyer, Convolution of orbital measures on symmetricspaces of type C p and D p , J. Aust. Math. Soc. 98, 232–256, 2015[4] P. Graczyk, P. Sawyer, On the product formula on noncompact Grass-mannians , Coll. Math. 133, 145–167, 2013.[5] P. Graczyk, P. Sawyer, A sharp criterion for the existence of the productformula on symmetric spaces of type A n , J. Lie Theory 20, 751–766, 2010.[6] P. Graczyk and P. Sawyer. On the kernel of the product formula on sym-metric spaces , Journal of Geometric Analysis, Vol. 14, 4, 653–672, 2004.[7] S. Helgason, Differential Geometry, Lie Groups and Symmetric spaces ,Graduate Studies in Mathematics, 34, American Mathematical Society,Providence, RI, 2001.[8] S. Helgason. Groups and geometric analysis. Integral geometry, invariantdifferential operators, and spherical functions , Mathematical Surveys andMonographs, 83, American Mathematical Society, Providence, RI, 2000.209] I. G. Macdonald. Commuting differential operators and zonal sphericalfunctions , Algebraic groups Utrecht, Lecture Notes in Mathematics 1271,(Springer-Verlag, New-York), 1987, 189–200.[10] Narayanana, E. K., A. Pasquale, A., and S. Pusti. Asymptotics ofHarish-Chandra expansions, bounded hypergeometric functions associ-ated with root systems, and applications , Advances in Mathematics, 252(2014), 227–259.[11] E. M. Opdam. Lecture notes on Dunkl operators for real and complexreflection groups. With a preface by Toshio Oshima , MSJ Memoirs, 8.Mathematical Society of Japan, Tokyo, 2000.[12] R. Rado. An inequality , J. London Math. Soc. 27, (1952), 1–6.[13] C. Rejeb. Harmonic and subharmonic functions associated to root sys-tems , Mathematics [math]. Universit´e Fran¸cois-Rabelais de Tours, Uni-versit´e de Tunis El Manar, 2015.[14] M. R¨osler. Positivity of Dunkls intertwining operator , Duke Math. J. 98(1999), no. 3, 445–463.[15] M. R¨osler. Dunkl operators: Theory and applications , Orthogonal Poly-nomials and Special Functions, Vol. 1817, Lecture Notes in Mathematics,93–135, 2003.[16] M. R¨osler and M. Voit. Positivity of Dunkl’s intertwining operator viathe trigonometric setting , Int. Math. Res. Not. 2004, no. 63, 3379–3389.[17] P. Sawyer. Spherical functions on symmetric cones , Trans. Amer. Math.Soc. 349 (1997), no. 9, 3569–3584.[18] P. Sawyer. The eigenfunctions of a Schr¨odinger operator associated tothe root system A n Quart. J. Math. Oxford Ser. (2) 50 (1999), no. 197,71–86.[19] K. Trim`eche. The positivity of the transmutation operators associatedwith the Cherednik operators attached to the root system of type A , Adv.Pure Appl. Math. 6 (2015), no. 2, 125–134.[20] L. W. Tu, An introduction to Manifolds , Second edition, Universitext,Springer, New York, 2011. 2121] H. Weyl,
, φ ( m ) λ ( e X ) are the generalized sphericalfunctions associated to the root system A n − as described in Definition 2.2. We will assume ℜ m > emark 2.1. It is readily seen that φ λ + a tr ( e H ) = e i a tr H φ λ ( e H ) for a ∈ C and φ λ ( e H + b I ) = e i b P nk =1 λ k φ λ ( e H ) for b ∈ R using induction. Our next step is to extend Theorem 2.1 to an arbitrary X ∈ a usingexpression (6). We start with a definition and notation. Definition 2.3.