Convex cocompactness for Coxeter groups
Jeffrey Danciger, François Guéritaud, Fanny Kassel, Gye-Seon Lee, Ludovic Marquis
CCONVEX COCOMPACTNESS FOR COXETER GROUPS
JEFFREY DANCIGER, FRANÇOIS GUÉRITAUD, FANNY KASSEL, GYE-SEON LEE,AND LUDOVIC MARQUIS
Dedicated to the memory of Ernest Borisovich Vinberg (1937–2020) A BSTRACT . We investigate representations of Coxeter groups into GL( n , (cid:82) ) as geomet-ric reflection groups which are convex cocompact in the projective space (cid:80) ( (cid:82) n ). Wecharacterize which Coxeter groups admit such representations, and we fully describethe corresponding spaces of convex cocompact representations as reflection groups, interms of the associated Cartan matrices. The Coxeter groups that appear include allinfinite, word hyperbolic, irreducible Coxeter groups; for such groups the representa-tions as reflection groups that we describe are exactly the projective Anosov ones. Wealso obtain a large class of nonhyperbolic irreducible Coxeter groups, thus providingmany examples for the theory of nonhyperbolic convex cocompact subgroups in (cid:80) ( (cid:82) n )developed in [DGK2]. C ONTENTS
1. Introduction 12. Reminders: Actions on properly convex subsets of projective space 103. Reminders: Vinberg’s theory for Coxeter groups 144. Maximality of the Vinberg domain 235. The minimal invariant convex subset of the Vinberg domain 276. Proof of Theorem 1.8 337. Proof of Theorem 1.3 and consequences 388. The deformation space of convex cocompact representations 40Appendix A. The spherical and affine Coxeter diagrams 46References 471. I
NTRODUCTION
In the setting of semisimple Lie groups of real rank one, convex cocompact subgroupsare a well-studied middle-ground between the very special class of cocompact latticesand the general class of all finitely generated discrete subgroups. The actions of suchgroups on the associated Riemannian symmetric space and its visual boundary at in-finity are known to exhibit many desirable geometric and dynamical properties.
Mathematics Subject Classification.
Key words and phrases.
Coxeter groups, reflection groups, discrete subgroups of Lie groups, convexcocompact subgroups, Anosov representations. a r X i v : . [ m a t h . G R ] F e b J. DANCIGER, F. GUÉRITAUD, F. KASSEL, G.-S. LEE, AND L. MARQUIS
In the setting of higher-rank semisimple Lie groups, however, it is not obvious howto generalize the notion of convex cocompact subgroup in a way that guarantees sim-ilar desirable properties while at the same time being flexible enough to admit largeinteresting classes of examples. In particular, the most natural generalization of con-vex cocompactness, in higher-rank Riemannian symmetric spaces, turns out to be quiterestrictive, see [KL, Q].In [DGK2], the first three authors investigated several notions of convex cocompact-ness for discrete subgroups of the higher-rank semisimple Lie group PGL( V ) actingon the projective space (cid:80) ( V ), where V is a finite-dimensional real vector space. Thisgeneralized both the classical theory of convex cocompactness in real hyperbolic ge-ometry, and the theory of divisible convex sets , as developed in particular by Benoist(see [Be5]). These notions of convex cocompactness in (cid:80) ( V ) come with a rich world ofexamples: see e.g. [Me, Be4, Ma1, BM, CM, Ba, CL, Ma2, BDL, CLM1, DGK1, DGK2,Z, LM, DGK3, CLM2]. The goal of the present paper is to study these notions in thecontext of Vinberg’s theory of discrete reflection groups.Reflection groups are a major source of interesting examples in the study of Kleiniangroups and hyperbolic geometry. While cocompact reflection groups in the real hyper-bolic space (cid:72) m may exist only for dimensions m (cid:201)
29 (see [Vi2]), there are rich familiesof convex cocompact reflection groups in (cid:72) m for any dimension m (cid:202) V ), giving conditions characterizing the discreteness ofa group generated by linear reflections in the hyperplanes bounding a convex poly-hedral cone in V . We will call these discrete groups reflection groups . When in-finite, they naturally identify with discrete subgroups of PGL( V ) via the projectionGL( V ) → PGL( V ). Vinberg’s construction includes the reflection groups in (cid:72) m as a spe-cial case for which the reflections preserve a nondegenerate quadratic form of signature( m , 1) on V . However, the construction also gives large families of interesting discretesubgroups of GL( V ) which are not contained in O( m , 1), including some that preservenondegenerate quadratic forms of signature other than ( m , 1), and many others thatdo not preserve any nonzero quadratic form at all. This is a rich source of examples ofdiscrete subgroups of infinite covolume in higher-rank semisimple Lie groups (see e.g.[KV, Be3, Be4, CLM1, LM]).The goal of the paper is to give an explicit characterization of the notions of con-vex cocompactness from [DGK2] in the setting of Vinberg’s reflection groups, with anapplication to the study of Anosov representations of word hyperbolic Coxeter groups.1.1. Convex cocompactness in projective space.
Let V be a real vector space ofdimension n (cid:202)
2. We say that an open subset Ω of the projective space (cid:80) ( V ) is convex if it is contained and convex in some affine chart, and properly convex if it is convex Vinberg referred to these groups by the name discrete linear groups generated by reflections , or simply linear Coxeter groups [Vi1, Def. 2].
ONVEX COCOMPACTNESS FOR COXETER GROUPS 3 and bounded in some affine chart. Here are the three notions of projective convexcocompactness from [DGK2] on which we focus.
Definition 1.1.
An infinite discrete subgroup Γ of GL( V ) is(i) naively convex cocompact in (cid:80) ( V ) if it acts properly discontinuously on someproperly convex open subset Ω of (cid:80) ( V ) and cocompactly on some nonempty Γ -invariant closed convex subset C of Ω ;(ii) convex cocompact in (cid:80) ( V ) if (i) holds and C ⊂ Ω can be taken “large enough”, inthe sense that the closure of C in (cid:80) ( V ) contains all accumulation points of allpossible Γ -orbits Γ · y with y ∈ Ω ;(iii) strongly convex cocompact in (cid:80) ( V ) if (i) holds and Ω can be taken so that itsboundary ∂ Ω is C and strictly convex (i.e. does not contain any nontrivial pro-jective segment).We note that (iii) implies (ii), as all Γ -orbits of Ω have the same accumulation pointswhen Ω is strictly convex (Remark 2.1).The three notions of convex cocompactness in Definition 1.1 may seem quite simi-lar at first glance, and they are indeed equivalent for discrete subgroups of Isom( (cid:72) m ) ⊂ O( m , 1). However, the three notions admit a number of subtle differences in general. Inparticular, naive convex cocompactness is not always stable under small deformations(see [DGK2, Rem. 4.4.(b)] and [DGK3]), whereas convex cocompactness and strong con-vex cocompactness are [DGK2, Th. 1.17]. By [DGK2, Th. 1.15], an infinite discrete sub-group of GL( V ) is strongly convex cocompact in (cid:80) ( V ) if and only if it is word hyperbolicand convex cocompact in (cid:80) ( V ).In the case that C = Ω in Definition 1.1, we say that Γ divides Ω , and that Ω is divisi-ble . Divisible convex sets have been very much studied since the 1960s. Examples with Γ nonhyperbolic (or equivalently ∂ Ω not strictly convex — hence (ii) is satisfied but not(iii)) include the case that Ω is the projective model of the Riemannian symmetric spaceof SL( k , (cid:82) ) with k (cid:202)
3, and Γ a uniform lattice (see e.g. [Be5, § 2.4]). The first irreducibleand nonsymmetric examples with Γ nonhyperbolic were constructed by Benoist [Be4]for 4 (cid:201) dim V (cid:201)
7, taking Γ to be a reflection group. Later, further examples were foundin [Ma1, BDL, CLM1].In the current paper, we give a necessary and sufficient condition (Theorem 1.8)for a reflection group to be convex cocompact in (cid:80) ( V ) in the sense of Definition 1.1.As a consequence, in the setting of right-angled reflection groups, the three notionsof convex cocompactness in Definition 1.1 are equivalent, and right-angled convex co-compact reflection groups are always word hyperbolic (Corollary 1.13). For generalreflection groups, we prove that naive convex cocompactness is always equivalent toconvex cocompactness, but there exist many non-right-angled reflection groups whichare convex cocompact without being strongly convex cocompact in (cid:80) ( V ) (beyond theexamples of [Be4, Ma1, BDL, CLM1]); these groups are not hyperbolic anymore, butrelatively hyperbolic (Corollary 1.7). J. DANCIGER, F. GUÉRITAUD, F. KASSEL, G.-S. LEE, AND L. MARQUIS
Vinberg’s theory of reflection groups.
Let W S = (cid:173) s , . . . , s N | ( s i s j ) m i , j = ∀ (cid:201) i , j (cid:201) N (cid:174) be a Coxeter group with generating set S = { s , . . . , s N } , where m i , i = m i , j = m j , i ∈ {
2, 3, . . . , ∞ } for i (cid:54)= j . (By convention, ( s i s j ) ∞ = s i s j has infinite order inthe group W S .) Each generator s i has order two. For any subset S (cid:48) of S , we denoteby W S (cid:48) the subgroup of W S generated by S (cid:48) , which we call a standard subgroup of W S .Recall that W S is said to be right-angled if m i , j ∈ { ∞ } for all i (cid:54)= j , and irreducible if it cannot be written as the direct product of two nontrivial standard subgroups. Anirreducible Coxeter group is said to be spherical if it is finite, and affine if it is infiniteand virtually abelian. We refer to Appendix A for the list of all spherical and all affineirreducible Coxeter groups.In the whole paper, V will denote a finite-dimensional real vector space, and we shalluse the following terminology and notation. Definition 1.2.
A representation ρ : W S → GL( V ) is a representation of W S as a reflec-tion group in V if • each element ρ ( s i ) is a hyperplane reflection, of the form ( x (cid:55)→ x − α i ( x ) v i ) forsome linear form α i ∈ V ∗ and some vector v i ∈ V with α i ( v i ) = • the convex polyhedral cone (cid:101) ∆ : = { v ∈ V | α i ( v ) (cid:201) ∀ i } has nonempty interior (cid:101) ∆ ◦ ; • ρ ( γ ) · (cid:101) ∆ ◦ ∩ (cid:101) ∆ ◦ = (cid:59) for all γ ∈ W S (cid:224) { e } .In this case the matrix A = ( α i ( v j )) (cid:201) i , j (cid:201) N is called a Cartan matrix for ρ .We denote by Hom ref ( W S , GL( V )) the set of all representations of W S as a reflectiongroup in V . Note that any ρ ∈ Hom ref ( W S , GL( V )) is faithful and discrete. Vinberg[Vi1, Th. 2] showed that the group ρ ( W S ) preserves an open convex cone of V , namelythe interior (cid:101) Ω Vin of the union of all ρ ( W S )-translates of the fundamental polyhedralcone (cid:101) ∆ . The group ρ ( W S ) acts properly discontinuously on (cid:101) Ω Vin . The action of ρ ( W S )is still properly discontinuous on the image Ω Vin ⊂ (cid:80) ( V ) of (cid:101) Ω Vin , which, for W S infiniteand irreducible, is a nonempty convex open subset of the projective space (cid:80) ( V ). Thecomposition of ρ with the projection GL( V ) → PGL( V ) is then still faithful and discrete.The pair ( α , v ) ∈ V ∗ × V defining a reflection is unique up to the action of (cid:82) ∗ by t · ( α , v ) = ( t α , t − v ). The reflections ρ ( s ), . . . , ρ ( s N ) therefore determine the matrix( α i ( v j )) (cid:201) i , j (cid:201) N up to conjugation by a diagonal matrix. In particular, they determinethe real numbers α i ( v j ) α j ( v i ). Vinberg [Vi1, Th. 1 & Prop. 6, 13, 17] gave the fol-lowing characterization, showing that Hom ref ( W S , GL( V )) is a semialgebraic subset ofHom( W S , GL( V )): a representation ρ : W S → GL( V ) belongs to Hom ref ( W S , GL( V )) if andonly if the α i and v j can be chosen so that the matrix A = ( A i , j ) (cid:201) i , j (cid:201) N = ( α i ( v j )) (cid:201) i , j (cid:201) N satisfies the following five conditions:(i) A i , j = i (cid:54)= j with m i , j = A i , j < i (cid:54)= j with m i , j (cid:54)= A i , j A j , i = ( π / m i , j ) for all i (cid:54)= j with 2 < m i , j < ∞ ;(iv) A i , j A j , i (cid:202) i (cid:54)= j with m i , j = ∞ ; and(v) (cid:101) ∆ : = { v ∈ V | α i ( v ) (cid:201) ∀ i } has nonempty interior. ONVEX COCOMPACTNESS FOR COXETER GROUPS 5
In that case, A is a Cartan matrix for ρ and is unique up to conjugation by positive diagonal matrices. If W S is infinite, irreducible, and not affine, then (v) is alwayssatisfied (see Sections 3.4–3.5). We shall prove (Proposition 4.1) that in this case theVinberg domain Ω Vin is maximal in the sense that it contains every ρ ( W S )-invariantproperly convex open subset of (cid:80) ( V ). We shall also describe (Theorem 5.2) the minimalnonempty ρ ( W S )-invariant properly convex open subset of (cid:80) ( V ).1.3. Coxeter groups admitting convex cocompact realizations.
Our first mainresult is a characterization of those Coxeter groups admitting representations as re-flection groups in V which are convex cocompact in (cid:80) ( V ). For this we consider thefollowing two conditions on W S : ¬ (IC) there do not exist disjoint subsets S (cid:48) , S (cid:48)(cid:48) of S such that W S (cid:48) and W S (cid:48)(cid:48) are bothinfinite and commute; ( (cid:101) A) for any subset S (cid:48) of S with S (cid:48) (cid:202)
3, if W S (cid:48) is irreducible and affine, then it is oftype (cid:101) A k where k = S (cid:48) − Theorem 1.3.
For an infinite irreducible Coxeter group W S , the following are equiva-lent:(1) there exist a finite-dimensional real vector space V and a representation ρ ∈ Hom ref ( W S , GL( V )) such that ρ ( W S ) is naively convex cocompact in (cid:80) ( V ) ;(2) W S satisfies conditions ¬ (IC) and ( (cid:101) A) .In this case, we can take V to be any vector space of dimension (cid:202) S, and ρ ( W S ) isactually convex cocompact in (cid:80) ( V ) (not only naively convex cocompact). Remark 1.4.
Conditions ¬ (IC) and ( (cid:101) A) are readily checked on the diagram of W S using the classification of spherical and affine Coxeter groups (see Appendix A). If W S satisfies ¬ (IC) and ( (cid:101) A) , then any standard subgroup W S (cid:48) of W S still satisfies ¬ (IC) and ( (cid:101) A) . Remark 1.5.
By a result of Krammer [Kr, Th. 6.8.3], conditions ¬ (IC) and ( (cid:101) A) to-gether are equivalent to the fact that any subgroup of W S isomorphic to (cid:90) is virtuallycontained in a conjugate of a standard subgroup W S (cid:48) of type (cid:101) A k for some k (cid:202) Remark 1.6.
Conditions ¬ (IC) and ( (cid:101) A) are always satisfied if W S is word hyper-bolic. In fact, Moussong [Mo] proved that word hyperbolicity of W S is equivalent tocondition ¬ (IC) together with ¬ (Af) there does not exist a subset S (cid:48) of S with S (cid:48) (cid:202) W S (cid:48) is irreducibleand affine.We recently used convex cocompact reflection groups in O( p , q ) to give a new proof ofthis hyperbolicity criterion: see [DGK1, Cor. 8.5] in the right-angled case and [LM, § 7]in the general case.Theorem 1.3, together with a result of Caprace [Ca1, Ca2], implies the following. Corollary 1.7.
Let W S be an infinite irreducible Coxeter group. If there exist V and ρ ∈ Hom ref ( W S , GL( V )) such that ρ ( W S ) is naively convex cocompact in (cid:80) ( V ) , then W S isrelatively hyperbolic with respect to a collection of virtually abelian subgroups of rank (cid:202) . These subgroups are the conjugates of the standard subgroups of W S of the form J. DANCIGER, F. GUÉRITAUD, F. KASSEL, G.-S. LEE, AND L. MARQUIS W U × W U ⊥ where W U is of type (cid:101) A k with k (cid:202) and W U ⊥ is the (finite) standard subgroupof W S generated by U ⊥ : = { s ∈ S (cid:224) U | m u , s = ∀ u ∈ U } . A characterization of convex cocompactness.
Our second main result is, fora Coxeter group W S as in Theorem 1.3, a simple characterization of convex cocompact-ness for representations ρ ∈ Hom ref ( W S , GL( V )). Theorem 1.8.
Let W S be an infinite irreducible Coxeter group satisfying conditions ¬ (IC) and ( (cid:101) A) of Section 1.3. For any V and any ρ ∈ Hom ref ( W S , GL( V )) with Cartanmatrix A = ( A i , j ) (cid:201) i , j (cid:201) N , the following are equivalent: (NCC) ρ ( W S ) is naively convex cocompact in (cid:80) ( V ) ; (CC) ρ ( W S ) is convex cocompact in (cid:80) ( V ) ; ¬ (ZT) for any irreducible standard subgroup W S (cid:48) of W S with (cid:59) (cid:54)= S (cid:48) ⊂ S, the Cartansubmatrix A W S (cid:48) : = ( A i , j ) s i , s j ∈ S (cid:48) is not of zero type; ¬ (ZD) det( A W S (cid:48) ) (cid:54)= for all S (cid:48) ⊂ S with W S (cid:48) of type (cid:101) A k , k (cid:202) .(The k = case of ¬ (ZD) just means that A i , j A j , i > for all i (cid:54)= j with m i , j = ∞ .) We refer to Definition 3.12 for the notion of zero type , and to Appendix A for type (cid:101) A k .In Theorem 1.8, the implication (CC) ⇒ (NCC) holds by definition; the equivalence ¬ (ZT) ⇔ ¬ (ZD) follows from classical results of Vinberg, see Fact 3.15 below. Weprove the other implications in Section 6. Remark 1.9.
In [Ma2], the last author studied groups generated by reflections in thecodimension-1 faces of a polytope ∆ ⊂ (cid:80) ( V ) which is (i.e. ∆ ∩ ∂ Ω Vin is a subsetof the vertices of ∆ ). In the case that the group acts strongly irreducibly on V , he gavea criterion [Ma2, Th. A] for naive convex cocompactness (Definition 1.1.(i)), as well asfor a notion of geometric finiteness, in terms of links of the vertices of the polytope.Here is an easy consequence of Theorem 1.8. Corollary 1.10.
For any infinite irreducible Coxeter group W S and any representation ρ ∈ Hom ref ( W S , GL( V )) , if ρ ( W S ) is convex cocompact in (cid:80) ( V ) , then so is ρ ( W S (cid:48) ) for anyinfinite standard subgroup W S (cid:48) of W S . Theorem 1.8 yields the following simple characterization of strong convex cocom-pactness for Coxeter groups.
Corollary 1.11.
Let W S be an infinite irreducible Coxeter group. For any V and any ρ ∈ Hom ref ( W S , GL( V )) with Cartan matrix A = ( A i , j ) (cid:201) i , j (cid:201) N , the following are equivalent: (SCC) ρ ( W S ) is strongly convex cocompact in (cid:80) ( V ) ; (WH+) W S is word hyperbolic and A i , j A j , i > for all i (cid:54)= j with m i , j = ∞ . Remark 1.12.
Corollary 1.11 generalizes a classical result stating that for ρ with val-ues in O( m , 1), the reflection group ρ ( W S ) is convex cocompact if and only if W S is wordhyperbolic and A i , j A j , i > i (cid:54)= j with m i , j = ∞ (see e.g. [DH, Th. 4.12]). ONVEX COCOMPACTNESS FOR COXETER GROUPS 7
It easily follows from Theorem 1.8 and Corollary 1.11 that in the setting of right-angled Coxeter groups, the three notions of convex cocompactness in Definition 1.1 areequivalent.
Corollary 1.13.
Let W S be an infinite irreducible Coxeter group with no standard sub-group of type (cid:101) A k for k (cid:202) , for instance an infinite irreducible right-angled Coxetergroup. For any V and any ρ ∈ Hom ref ( W S , GL( V )) with Cartan matrix A = ( A i , j ) (cid:201) i , j (cid:201) N ,the following are equivalent: (NCC) ρ ( W S ) is naively convex cocompact in (cid:80) ( V ) ; (CC) ρ ( W S ) is convex cocompact in (cid:80) ( V ) ; (SCC) ρ ( W S ) is strongly convex cocompact in (cid:80) ( V ) ; (WH+) W S is word hyperbolic and A i , j A j , i > for all i (cid:54)= j with m i , j = ∞ . Remark 1.14.
Corollary 1.13 was previously proved in the case that ρ preserves anondegenerate quadratic form on V by the first three authors when W S is right-angled[DGK1, Th. 8.2] and by the last two authors [LM, Th. 4.6] in general.1.5. The subspace of convex cocompact representations.
By [DGK2, Th. 1.17],the set of convex cocompact representations is open in Hom( W S , GL( V )), but can wecharacterize it more precisely?Suppose W S is word hyperbolic and let ρ ∈ Hom ref ( W S , GL( V )) have Cartan matrix A = ( A i , j ) (cid:201) i , j (cid:201) N . As we recalled in Section 1.2, the weak inequality A i , j A j , i (cid:202) m i , j = ∞ . By Corollary 1.13, ρ ( W S ) is convex cocompact in (cid:80) ( V ) if and onlyif these inequalities are all strict. It is then natural to ask if the open subset of convexcocompact representations is the full interior of Hom ref ( W S , GL( V )) in Hom( W S , GL( V )).We show that the answer is yes in large enough dimension (Corollary 1.16) but not ingeneral (Example 8.5).Recall that a representation ρ of a discrete group Γ into GL( V ) is said to be semisim-ple if it is a direct sum of irreducible representations. The orbit of a representation ρ under the action of GL( V ) by conjugation is closed if and only if ρ is semisimple.The quotient of the space of semisimple representations by GL( V ) is called the space ofcharacters of Γ in GL( V ), we denote it by χ ( Γ , GL( V )). If W S is a Coxeter group, then acharacter coming from a representation of W S as a reflection group in V is called a re-flection character , and we shall denote the space of such characters by χ ref ( W S , GL( V )).As a consequence of Corollary 1.13, we prove the following. Corollary 1.15.
Let W S be an infinite, word hyperbolic, irreducible Coxeter group.For any V with n : = dim V (cid:202) N : = S, the set of characters [ ρ ] ∈ χ ref ( W S , GL( V )) forwhich ρ ( W S ) is convex cocompact in (cid:80) ( V ) is precisely the interior of χ ref ( W S , GL( V )) in χ ( W S , GL( V )) . The main content of Corollary 1.15 is the case n = N . Indeed, a semisimple rep-resentation ρ ∈ Hom ref ( W S , GL( V )) with Cartan matrix A splits as a direct sum of arepresentation ρ (cid:48) of dimension rank( A ) plus a trivial representation of complementarydimension. When n (cid:202) N , the space χ ref ( W S , GL( V )) is homeomorphically parametrized J. DANCIGER, F. GUÉRITAUD, F. KASSEL, G.-S. LEE, AND L. MARQUIS by the space of equivalence classes of N × N Cartan matrices that are compatiblewith W S (see Definition 3.7). Hence the structure of χ ref ( W S , GL( V )) is the same forall n = dim( V ) (cid:202) N . Note that the bound n (cid:202) N in Corollary 1.15 cannot be improvedin general, since when n < N , the condition that rank( A ) (cid:201) n imposes a nontrivial con-straint so that the entries of the Cartan matrix may not be deformed freely. It canhappen that 4 is a local minimum value of the function A i , j A j , i for some pair of gener-ators with m i , j = ∞ (see Example 8.5). Note also that Corollary 1.15 does not extendto nonhyperbolic groups (see Example 8.6).The structure of Hom ref ( W S , GL( V )) is more complicated than that of χ ref ( W S , GL( V )).While a Cartan matrix A determines at most one conjugacy class of semisimple repre-sentations in Hom ref ( W S , GL( V )), there may be many conjugacy classes of nonsemisim-ple representations in Hom ref ( W S , GL( V )) with Cartan matrix A . Further, even incases when n (cid:202) N , it is difficult to determine if the map sending a representation ρ generated by reflections to the (equivalence class of a) Cartan matrix for ρ is open. Weprove the following. Corollary 1.16.
Let W S be an infinite, word hyperbolic, irreducible Coxeter group inN = S generators and let V be a real vector space of dimension n. If(1) n (cid:202) N − , or(2) W S is right-angled and n (cid:202) N,then the set of representations ρ ∈ Hom ref ( W S , GL( V )) for which ρ ( W S ) is convex cocom-pact in (cid:80) ( V ) is precisely the interior of Hom ref ( W S , GL( V )) in Hom( W S , GL( V )) . Corollary 1.16 is a consequence of the equivalence (CC) ⇔ (WH+) of Corollary 1.13together with some analysis of how the entries of the Cartan matrix deform under thedimension restriction (see Section 8.2).1.6. Anosov representations for Coxeter groups.
We give an application of Corol-lary 1.11 to the topic of
Anosov representations of word hyperbolic groups into GL( V ).These are representations with strong dynamical properties which generalize con-vex cocompact representations into rank-one semisimple Lie groups (see in particular[La, GW, KLP1, KLP2, GGKW, BPS]). They have been much studied recently, espe-cially in the setting of higher Teichmüller theory.We shall not recall the definition of Anosov representations in this paper, nor assumeany technical knowledge of them. We shall just use the following relation with convexcocompactness in (cid:80) ( V ), proven in [DGK1, DGK2] and also, in a slightly different form,in [Z]. Fact 1.17 ([DGK2, Th. 1.4]) . Let Γ be an infinite discrete subgroup of GL( V ) acting prop-erly discontinuously on some nonempty properly convex open subset Ω of (cid:80) ( V ) . Then Γ is strongly convex cocompact in (cid:80) ( V ) (Definition 1.1.(iii)) if and only if Γ is word hyper-bolic and the natural inclusion Γ (cid:44) → GL( V ) is P -Anosov (i.e. Anosov with respect to thestabilizer of a line in V ). ONVEX COCOMPACTNESS FOR COXETER GROUPS 9
This relation allows for the construction of new examples of discrete subgroupsof GL( V ) which are strongly convex cocompact in (cid:80) ( V ), by using classical examplesof Anosov representations: see e.g. [DGK2, Prop. 1.7] or [Z, Cor. 1.32]. Conversely,Fact 1.17 gives a way to obtain new examples of Anosov representations by construct-ing strongly convex cocompact groups: this is the point of view adopted in [DGK1, § 8]and in the present paper.More precisely, in [DGK1, § 8] and [LM, § 7] we constructed, for any infinite, word hy-perbolic, irreducible Coxeter group W S , examples of representations ρ ∈ Hom ref ( W S , GL( V ))which are convex cocompact in (cid:80) ( V ) for some V ; by Fact 1.17 this yielded examples of P -Anosov representations of W S (or any quasi-isometrically embedded subgroup) intoGL( V ). These representations took values in the orthogonal group of a nondegeneratequadratic form. Obtaining such representations is interesting because, while Anosovrepresentations of free groups and surface groups are abundant in the literature, ex-amples of Anosov representations of more complicated hyperbolic groups have beenmuch less commonly known outside the realm of Kleinian groups.Consider the set of P -Anosov representations of a word hyperbolic (not necessarilyright-angled) Coxeter group into GL( V ) for an arbitrary V . Corollaries 1.11 and 1.16,together with Fact 1.17, yield the following description of those Anosov representationsthat realize the group as a reflection group in V (see Section 8.3). Corollary 1.18.
Let W S be an infinite, word hyperbolic, irreducible Coxeter group. Forany V and any ρ ∈ Hom ref ( W S , GL( V )) with Cartan matrix A = ( A i , j ) (cid:201) i , j (cid:201) N , the follow-ing are equivalent: • ρ is P -Anosov; • A i , j A j , i > for all i (cid:54)= j with m i , j = ∞ .If moreover n (cid:202) N − , or W S is right-angled and n (cid:202) N, where n = dim V and N = S,then the space of P -Anosov representations of W S as a reflection group is precisely theinterior of Hom ref ( W S , GL( V )) in Hom( W S , GL( V )) . Remark 1.19. If n < N , then the space of P -Anosov representations of W S as a reflec-tion group may be smaller than the interior of Hom ref ( W S , GL( V )). This is the case inExample 8.5 below, where W S is word hyperbolic and the interior of Hom ref ( W S , GL( V ))contains a unique representation which is faithful and discrete but not Anosov. Theexistence of such a non-Anosov representation of a word hyperbolic group Γ admittinga neighborhood in Hom( Γ , GL( V )) consisting entirely of faithful and discrete represen-tations provides a negative answer to a question asked in [Kas, § 8] and [P, § 4.3].1.7. Organization of the paper.
In Section 2 we recall some well-known facts in con-vex projective geometry, as well as some results from [DGK1, DGK2]. In Section 3 werecall Vinberg’s theory of linear reflection groups and provide proofs of some basic re-sults. In Section 4 we establish the maximality of the Vinberg domain (Proposition 4.1),and in Section 5 we describe the minimal invariant convex domain of a reflection group.In Section 6 we prove Theorem 1.8, and in Section 7 we establish Theorem 1.3 andCorollaries 1.7, 1.10, 1.11, and 1.13. In Section 8 we conclude with the proofs of Corol-laries 1.15, 1.16, and 1.18, and give examples showing that the dimension condition inCorollary 1.15 is optimal.
Acknowledgements.
Upon hearing about the first three authors’ results on the right-angled case, Mike Davis kindly shared with them some personal notes he had beengathering on the topic over the past few years. We would like to thank him for hiskeen interest. The last two authors are especially grateful to Ryan Greene, since theystarted thinking about this topic with him several years ago. We also thank AnnaWienhard for helpful conversations.This project received funding from the European Research Council (ERC) under theEuropean Union’s Horizon 2020 research and innovation programme (ERC startinggrant DiGGeS, grant agreement No 715982, and ERC consolidator grant Geometric-Structures, grant agreement No 614733). The authors also acknowledge support fromthe GEAR Network, funded by the National Science Foundation under grants DMS1107452, 1107263, and 1107367 (“RNMS: GEometric structures And Representationvarieties”). J.D. was partially supported by an Alfred P. Sloan Foundation fellowshipand by the National Science Foundation under grants DMS 1510254, DMS 1812216,and DMS 1945493. F.G. and F.K. were partially supported by the Agence Nationalede la Recherche through the grant DynGeo (ANR-16-CE40-0025-01) and the LabexCEMPI (ANR-11-LABX-0007-01), and F.G. through the Labex IRMIA (11-LABX-0055).Part of this work was completed while F.K. was in residence at the MSRI in Berkeley,California, for the program
Geometric Group Theory (Fall 2016) supported by NSFgrant DMS 1440140, and at the INI in Cambridge, UK, for the program
Nonposi-tive curvature group actions and cohomology (Spring 2017) supported by EPSRC grantEP/K032208/1. G.L. was partially supported by the DFG research grant “Higher Teich-müller Theory” and by the National Research Foundation of Korea (NRF) grant fundedby the Korean government (MSIT) (No 2020R1C1C1A01013667). L.M. acknowledgessupport by the Centre Henri Lebesgue (ANR-11-LABX-0020 LEBESGUE).2. R
EMINDERS : A
CTIONS ON PROPERLY CONVEX SUBSETS OF PROJECTIVE SPACE
In the whole paper, we fix a real vector space V of dimension n (cid:202)
2. We first recallsome well-known facts in convex projective geometry, as well as some results from[DGK1, DGK2]. Here we work with the projective general linear group PGL( V ), whichnaturally acts on the projective space (cid:80) ( V ); in the rest of the paper, we shall workwith subgroups of GL( V ), which also acts on (cid:80) ( V ) via the projection GL( V ) → PGL( V ).All of the discrete subgroups of GL( V ) that we consider project injectively to discretesubgroups of PGL( V ).2.1. Properly convex domains in projective space.
Recall that an open domain Ω in the projective space (cid:80) ( V ) is said to be convex if it is contained and convex insome affine chart, properly convex if it is convex and bounded in some affine chart, and strictly convex if in addition its boundary does not contain any nontrivial projectiveline segment. It is said to have C boundary if every point of the boundary of Ω has aunique supporting hyperplane. ONVEX COCOMPACTNESS FOR COXETER GROUPS 11
Let Ω be a properly convex open subset of (cid:80) ( V ), with boundary ∂ Ω . Recall the Hilbertmetric d Ω on Ω : d Ω ( y , z ) : =
12 log [ a , y , z , b ]for all distinct y , z ∈ Ω , where [ · , · , · , · ] is the cross-ratio on (cid:80) ( (cid:82) ), normalized so that[0, 1, z , ∞ ] = z , and where a , b are the intersection points of ∂ Ω with the projective linethrough y and z , with a , y , z , b in this order (see Figure 1). The metric space ( Ω , d Ω )is complete (i.e. Cauchy sequences converge) and proper (i.e. closed balls are compact),and the group Aut( Ω ) : = { g ∈ PGL( V ) | g · Ω = Ω } acts on Ω by isometries for d Ω . As a consequence, any discrete subgroup of Aut( Ω ) actsproperly discontinuously on Ω . y za b Ω F IGURE
1. Hilbert distanceLet V ∗ be the dual vector space of V . By definition, the dual convex set of Ω is Ω ∗ : = (cid:80) (cid:161)(cid:169) ϕ ∈ V ∗ | ϕ ( x ) < ∀ x ∈ (cid:101) Ω (cid:170)(cid:162) ,where (cid:101) Ω is the closure in V (cid:224) { } of an open convex cone of V lifting Ω . The set Ω ∗ is aproperly convex open subset of (cid:80) ( V ∗ ) which is preserved by the dual action of Aut( Ω )on (cid:80) ( V ∗ ).Straight lines (contained in projective lines) are always geodesics for the Hilbertmetric d Ω . When Ω is not strictly convex, there may be other geodesics as well. How-ever, a biinfinite geodesic of ( Ω , d Ω ) always has well-defined, distinct endpoints in ∂ Ω ,by [FK, Th. 3] (see also [DGK1, Lem. 2.6]). Remark 2.1. If Ω is strictly convex, then all Γ -orbits of Ω have the same accumulationpoints in ∂ Ω . Indeed, let y ∈ Ω and ( γ n ) ∈ Γ (cid:78) be such that ( γ n · y ) converges to some y ∞ ∈ ∂ Ω . Let z be another point of Ω . After possibly passing to a subsequence, ( γ n · z )converges to some z ∞ ∈ ∂ Ω . By properness of the action of Γ on Ω , the limit [ y ∞ , z ∞ ]of the sequence of compact intervals ( γ n · [ y , z ]) is contained in ∂ Ω . Strict convexitythen implies that y ∞ = z ∞ . This shows that any accumulation point of Γ · y is also anaccumulation point of Γ · z .2.2. The proximal limit set and the full orbital limit set.
Recall that an elementof PGL( V ) is called proximal in (cid:80) ( V ) if it has a unique eigenvalue of largest modulusand this eigenvalue has multiplicity 1. We shall use the following terminology as in[DGK1, DGK2]. Definition 2.2.
Let Γ be a discrete subgroup of PGL( V ). The proximal limit set of Γ in (cid:80) ( V ) is the closure Λ Γ of the set of attracting fixed points of elements of Γ which areproximal in (cid:80) ( V ).When Γ acts irreducibly on (cid:80) ( V ), the proximal limit set Λ Γ was studied in [Gu, Be1,Be2]. In that setting, the action of Γ on Λ Γ is minimal (i.e. any orbit is dense) and Λ Γ is contained in any nonempty closed Γ -invariant subset of (cid:80) ( V ) [Be1, Lem. 3.6].We shall use the following terminology from [DGK2]. Definition 2.3.
Let Γ be an infinite discrete subgroup of PGL( V ) preserving a nonemptyproperly convex open subset Ω of (cid:80) ( V ). • The full orbital limit set of Γ in Ω is the set Λ orb Ω ( Γ ) of all points which are anaccumulation point in ∂ Ω of some Γ -orbit Γ · y ⊂ Ω for y ∈ Ω . • The convex core of Γ in Ω is the convex hull C cor Ω ( Γ ) of Λ orb Ω ( Γ ) in Ω . • The action of Γ on Ω is convex cocompact if Γ acts cocompactly on C cor Ω ( Γ ).Thus an infinite discrete subgroup Γ of PGL( V ) is convex cocompact in (cid:80) ( V ) (Defini-tion 1.1.(ii)) if and only if it acts convex cocompactly (Definition 2.3) on some properlyconvex open subset Ω ⊂ (cid:80) ( V ). In this case the proximal limit set Λ Γ is always nonemptyIf Γ acts convex cocompactly (e.g. divides) a properly convex open subset Ω of (cid:80) ( V ),then the proximal limit set Λ Γ is always nonempty: see [DGK2]. In the proof of Propo-sition 6.5 we shall use the following classical fact. Fact 2.4 ([Ve, Prop. 3]) . Let Γ be a discrete subgroup of PGL( V ) dividing (i.e. actingproperly discontinuously and cocompactly on) some properly convex open subset Ω of (cid:80) ( V ) . Then the convex hull of Λ Γ in Ω is equal to Ω . Maximal and minimal domains.
We shall use the following properties, whichare due to Benoist [Be2] in the irreducible case. We denote by Z the closure of a set Z in (cid:80) ( V ) or (cid:80) ( V ∗ ). Proposition 2.5 (see [Be2, Prop. 3.1] and [DGK2, Prop. 3.10]) . Let Γ be a discrete sub-group of PGL( V ) preserving a nonempty properly convex open subset Ω of (cid:80) ( V ) andcontaining an element which is proximal in (cid:80) ( V ) . Let Λ Γ (resp. Λ ∗ Γ ) be the proximallimit set of Γ in (cid:80) ( V ) (resp. (cid:80) ( V ∗ ) ). Then(1) Λ Γ ⊂ ∂ Ω ∩ Γ · y for all y ∈ Ω , and Λ ∗ Γ ⊂ ∂ Ω ∗ ∩ Γ · [ ϕ ] for all [ ϕ ] ∈ Ω ∗ ;(2) more specifically, Ω and Λ Γ lift to cones (cid:101) Ω and (cid:101) Λ Γ of V (cid:224) { } with (cid:101) Ω properlyconvex containing (cid:101) Λ Γ in its boundary, and Ω ∗ and Λ ∗ Γ lift to cones (cid:101) Ω ∗ and (cid:101) Λ ∗ Γ of V ∗ (cid:224) { } with (cid:101) Ω ∗ properly convex containing (cid:101) Λ ∗ Γ in its boundary, such that ϕ ( x ) (cid:202) for all x ∈ (cid:101) Λ Γ and ϕ ∈ (cid:101) Λ ∗ Γ ;(3) for (cid:101) Λ ∗ Γ as in (2), the set Ω max : = (cid:80) (cid:161)(cid:169) x ∈ V | ϕ ( x ) > ∀ ϕ ∈ (cid:101) Λ ∗ Γ (cid:170)(cid:162) is the connected component of (cid:80) ( V ) (cid:224) (cid:83) [ ϕ ] ∈ Λ ∗ Γ (cid:80) Ker( ϕ ) containing Ω ; it is Γ -invariant,convex, and open in (cid:80) ( V ) ; any Γ -invariant properly convex open subset Ω (cid:48) of (cid:80) ( V ) containing Ω is contained in Ω max . ONVEX COCOMPACTNESS FOR COXETER GROUPS 13
In (3), the convex set Ω max is not always properly convex. However, in the irreduciblecase it is: Proposition 2.6 ([Be2, Prop. 3.1]) . Let Γ be a discrete subgroup of PGL( V ) preservinga nonempty properly convex open subset Ω of (cid:80) ( V ) .If Γ acts irreducibly on (cid:80) ( V ) , then Γ always contains a proximal element and theset Ω max of Proposition 2.5. (3) is always properly convex. Moreover, there is a smallestnonempty Γ -invariant convex open subset Ω min of Ω max , namely the interior of the convexhull of Λ Γ in Ω max .If moreover Γ acts strongly irreducibly on (cid:80) ( V ) (i.e. all finite-index subgroups of Γ actirreducibly), then Ω max is the unique maximal Γ -invariant properly convex open set in (cid:80) ( V ) ; it contains all other Γ -invariant properly convex open subsets. Convex cocompactness in projective space.
Recall the notions of naive con-vex cocompactness, convex cocompactness, and strong convex cocompactness from Def-inition 1.1, as well as the notion of convex cocompact action from Definition 2.3. Weshall use the following characterizations and properties from [DGK2].
Proposition 2.7. [DGK2, Th. 1.15]
Let Γ be an infinite discrete subgroup of PGL( V ) .Then Γ is strongly convex cocompact in (cid:80) ( V ) if and only if it is convex cocompact in (cid:80) ( V ) and word hyperbolic. Proposition 2.8 ([DGK2, Prop. 11.4]) . Let Γ be an infinite discrete subgroup of PGL( V ) .If Γ is naively convex cocompact in (cid:80) ( V ) , then it does not contain any unipotent element. Proposition 2.9 ([DGK2, Th. 1.17.(A)]) . Let Γ be an infinite discrete subgroup of PGL( V ) .The group Γ is convex cocompact in (cid:80) ( V ) if and only if it is convex cocompact in (cid:80) ( V ∗ ) (for the dual action). Proposition 2.10 ([DGK2, Prop. 11.9]) . Let Γ be an infinite discrete subgroup of SL ± ( V ) acting trivially on some linear subspace V of V . The group Γ is convex cocompact in (cid:80) ( V ) if and only if the induced representation Γ → SL ± ( V / V ) is injective and its imageis convex cocompact in (cid:80) ( V / V ) . The last two statements imply the following.
Corollary 2.11.
Let Γ be an infinite discrete subgroup of SL ± ( V ) preserving a linearsubspace V of V , such that the induced action on V / V is trivial. The group Γ is convexcocompact in (cid:80) ( V ) if and only if the induced representation Γ → SL ± ( V ) is injective andits image is convex cocompact in (cid:80) ( V ) .Proof. For any subspace V (cid:48) of V , we denote by Ann( V (cid:48) ) ⊂ V ∗ the annihilator of V (cid:48) , i.e.the subspace of linear forms ϕ ∈ V ∗ that vanish on V (cid:48) . The dual V (cid:48)∗ identifies with V ∗ /Ann( V (cid:48) ).Let V be a complementary subspace of V in V . In a basis adapted to the decompo-sition V = V ⊕ V , the group Γ is expressed as a group of matrices of the form (cid:161) i ( γ ) ∗ (cid:162) ,where i : Γ → SL ± ( V ) is the restricted representation. In the dual basis, which is adapted to the decomposition V ∗ = Ann( V ) ⊕ Ann( V ), the dual action of Γ on V ∗ isgiven by (cid:161) t i ( γ ) − ∗ Id (cid:162) . In other words, Γ acts trivially on the subspace Ann( V ) of V ∗ .By Proposition 2.10, the group Γ is convex cocompact in (cid:80) ( V ∗ ) if and only if the in-duced representation γ (cid:55)→ t i ( γ ) − of Γ is injective and its image is convex cocompact in (cid:80) ( V ∗ /Ann( V )) (cid:39) (cid:80) ( V ∗ ). Dualizing again, by Proposition 2.9, the group Γ is convex co-compact in (cid:80) ( V ) if and only if the restricted representation i is injective and its imageis convex cocompact in (cid:80) ( V ). (cid:3)
3. R
EMINDERS : V
INBERG ’ S THEORY FOR C OXETER GROUPS
In this section we set up some notation and recall the basics of Vinberg’s theory [Vi1]of linear reflection groups.3.1.
Coxeter groups.
Let W S be a Coxeter group generated by a finite set of involu-tions S = { s , . . . , s N } , with presentation(3.1) W S = (cid:173) s , . . . , s N | ( s i s j ) m i , j = ∀ (cid:201) i , j (cid:201) N (cid:174) where m i , i = m i , j = m j , i ∈ {
2, 3, . . . , ∞ } for i (cid:54)= j . The Coxeter diagram for W S is alabeled graph G W such that(i) the set of nodes (i.e. vertices) of G W is the set S ;(ii) two nodes s i , s j ∈ S are connected by an edge s i s j of G W if and only if m i , j ∈ {
3, 4, . . . , ∞ } ;(iii) the label of the edge s i s j is m i , j .It is customary to omit the label of the edge s i s j if m i , j = S (cid:48) of S , the subgroup W S (cid:48) of W S generated by S (cid:48) is the Coxeter groupwith generating set S (cid:48) and exponents m (cid:48) i , j = m i , j for s i , s j ∈ S (cid:48) , see [Bo, Chap. IV, Th. 2].Such a subgroup W S (cid:48) is called a standard subgroup of W S .The connected components of the graph G W are graphs of the form G W Si , i ∈ I , wherethe S i form a partition of S . The subsets S i are called the irreducible components of S .The group W S is the direct product of the standard subgroups W S i for i ∈ I .We shall assume that W S is irreducible , which means that G W is connected, or inother words that S has only one irreducible component ( I = W S will furthermore always beinfinite.3.2. Representing the generators of W S by reflections in hyperplanes. We shalluse the following terminology.
ONVEX COCOMPACTNESS FOR COXETER GROUPS 15
Definition 3.1.
We call an N × N real matrix A = ( A i , j ) (cid:201) i , j (cid:201) N weakly compatible withthe Coxeter group W S if(3.2) A i , i = i , A i , j = i (cid:54)= j with m i , j = A i , j < i (cid:54)= j with m i , j (cid:54)= A i , j A j , i = ( π / m i , j ) for all i (cid:54)= j with 2 < m i , j < ∞ .Consider N -tuples α = ( α , . . . , α N ) ∈ V ∗ N of linear forms and v = ( v , . . . , v N ) ∈ V N ofvectors. If the matrix A = (cid:161) α i ( v j ) (cid:162) (cid:201) i , j (cid:201) N is weakly compatible, then for any i ,(3.3) ρ ( s i ) : = (cid:161) x (cid:55)−→ x − α i ( x ) v i (cid:162) is a linear reflection of V in the hyperplane Ker( α i ), and (3.3) defines a group homo-morphism ρ : W S → GL( V ). Definition 3.2.
A representation ρ : W S → GL( V ) is generated by weakly compatiblereflections if it is defined by (3.3) for some α = ( α , . . . , α N ) ∈ V ∗ N and v = ( v , . . . , v N ) ∈ V N with A = (cid:161) α i ( v j ) (cid:162) (cid:201) i , j (cid:201) N weakly compatible with W S . In this case we say that A isthe Cartan matrix of ( α , v ), and a Cartan matrix for ρ .We note that for any i , the pair ( α i , v i ) ∈ V ∗ × V defining the reflection ρ ( s i ) in (3.3)is not uniquely determined by ρ ( s i ). Indeed, for any λ i > λ i α i , λ − i v i ) yieldsthe same reflection. Changing ( α i , v i ) into ( λ i α i , λ − i v i ) changes the Cartan matrix A into its conjugate D A D − where D = diag( λ , . . . , λ N ) is diagonal with positive entries. Definition 3.3.
Two N × N matrices which are weakly compatible with W S are consid-ered equivalent if they differ by conjugation by a positive diagonal matrix.While the Cartan matrix A is not uniquely determined by the representation ρ , itsequivalence class is. In particular, for any 1 (cid:201) i , j (cid:201) N , the product A i , j A j , i is uniquelydetermined by ρ . The quantity A i , j A j , i varies as an algebraic function, invariant underconjugation, as ρ ranges over the semialgebraic set of representations of W S to GL( V )generated by weakly compatible reflections. Remark 3.4.
There exist representations of W S generated by linear reflections whichare not weakly compatible with W S : for instance, when the matrix ( α i ( v j )) (cid:201) i , j (cid:201) N satis-fies (3.2) with π replaced by 2 π . The representations by weakly compatible reflectionsform an open subset of Hom( W S , GL( V )), containing the set Hom ref ( W S , GL( V )) of Sec-tions 1.2 and 3.3 below.The following notation and terminology will be used throughout the rest of the paper. Definition 3.5.
Let ρ : W S → GL( V ) be a representation generated by weakly compat-ible reflections, associated to some α = ( α , . . . , α N ) ∈ V ∗ N and v = ( v , . . . , v N ) ∈ V N . Let V v be the span of v , . . . , v N , and V α the intersection of the kernels of α , . . . , α N . We saythat the representation ρ (or its image ρ ( W S )) is reduced if V α = { } , and dual-reduced if V v = V . Remark 3.6.
In general, a representation ρ : W S → GL( V ) generated by weakly com-patible reflections induces three representations: • ρ v on V v which is a dual-reduced representation, • ρ α on V α : = V / V α which is a reduced representation, • ρ α v on V α v : = V v /( V v ∩ V α ) which is a reduced and dual-reduced representation.These three representations are still generated by weakly compatible reflections withthe same Cartan matrix A = ( α i ( v j )) (cid:201) i , j (cid:201) N .3.3. Representations of W S as a reflection group. Let ρ : W S → GL( V ) be a repre-sentation of W S generated by weakly compatible reflections (Definition 3.2), associatedto some α = ( α , . . . , α N ) ∈ V ∗ N and v = ( v , . . . , v N ) ∈ V N ; by definition, the Cartan ma-trix A = ( α i ( v j )) (cid:201) i , j (cid:201) N satisfies (3.2). The cone(3.4) (cid:101) ∆ : = (cid:169) x ∈ V | α i ( x ) (cid:201) ∀ (cid:201) i (cid:201) N (cid:170) is called the closed fundamental polyhedral cone in V associated to ( α , v ). As mentionedin Section 1.2, assuming(3.5) the convex polyhedral cone (cid:101) ∆ has nonempty interior (cid:101) ∆ ◦ ,Vinberg [Vi1] proved that this cone has N sides, and that ρ ( γ ) · (cid:101) ∆ ◦ ∩ (cid:101) ∆ ◦ = (cid:59) for all γ ∈ W S (cid:224) { e } if and only if the Cartan matrix A further satisfies(3.6) A i , j A j , i (cid:202) i (cid:54)= j with m i , j = ∞ .In that case, ρ is injective, and the stabilizer of any face of (cid:101) ∆ coincides with the stan-dard subgroup generated by the reflections in the hyperplanes Ker( α i ) containing thatface. Definition 3.7.
We call an N × N real matrix A = ( A i , j ) (cid:201) i , j (cid:201) N compatible with the Cox-eter group W S if A is weakly compatible with W S (Definition 3.1) and satisfies (3.6).A representation ρ : W S → GL( V ) generated by reflections in hyperplanes such thatthe Cartan matrix A = ( α i ( v j )) (cid:201) i , j (cid:201) N is compatible and such that (3.5) holds is a repre-sentation of W S as a reflection group in V as in Definition 1.2. We emphasize that (3.5)always holds if W S is irreducible and nonaffine (see Remark 3.13). As in Section 1.2,we denote by Hom ref ( W S , GL( V )) the set of representations of W S as a reflection groupin V .Consider ρ ∈ Hom ref ( W S , GL( V )). By [Vi1, Th. 2], the group ρ ( W S ) acts properly dis-continuously on a nonempty convex open cone (cid:101) Ω Vin of V , namely the interior of theunion of all ρ ( W S )-translates of the fundamental polyhedral cone (cid:101) ∆ . Fact 3.8 ([Vi1, Th. 2]) . The set (cid:101) ∆ (cid:91) : = (cid:101) ∆ ∩ (cid:101) Ω Vin is equal to (cid:101) ∆ minus its faces of infinitestabilizer; it is a fundamental domain for the action of ρ ( W S ) on (cid:101) Ω Vin . The cone (cid:101) Ω Vin , sometimes called the
Tits cone , projects to a nonempty ρ ( W S )-invariantopen subset Ω Vin of (cid:80) ( V ), which is convex whenever W S is infinite and irreducible; weshall call it the Tits–Vinberg domain (or simply the
Vinberg domain ). A fundamentaldomain ∆ (cid:91) for the action on Ω Vin is obtained from the projection ∆ of (cid:101) ∆ to (cid:80) ( V ) againby removing the faces of infinite stabilizer. ONVEX COCOMPACTNESS FOR COXETER GROUPS 17
Remark 3.9.
In fact, [Vi1, Th. 2] shows that the union of all ρ ( W S )-translates of (cid:101) ∆ (notonly of (cid:101) ∆ (cid:91) ) is also convex. Example 3.10 (see [Vi1, § 2] and Figure 2) . Suppose N = m = ∞ . Then W S (cid:39) ( (cid:90) /2 (cid:90) ) ∗ ( (cid:90) /2 (cid:90) ) is of type (cid:101) A (see Appendix A). The infinite cyclic subgroup generatedby s s has index two in W S .(i) Suppose t = A A >
4. Then v , v are linearly independent and V = V v ⊕ V α .The polyhedral cone (cid:101) ∆ is the set(3.7) (cid:189) λ v + λ v (cid:175)(cid:175)(cid:175) λ , λ > | A | (cid:201) λ λ (cid:201) | A | (cid:190) + V α .The element ρ ( s s ) acts on V α trivially, and on V v as the matrix (cid:161) − + A A A − A − (cid:162) (in thebasis ( v , v )). In particular, ρ ( s s ) and ρ ( s s ) − are proximal in (cid:80) ( V ), with attractingfixed points [ x + ], [ x − ] ∈ (cid:80) ( V ) where x ± = ( t ± (cid:112) t ( t − v − A v , and (cid:101) Ω Vin = (cid:82) > x + + (cid:82) > x − + V α .Since the induced action of ρ ( W S ) on the image of (cid:101) Ω Vin in (cid:80) ( V / V α ) is cocompact andthe induced representation Γ → SL ± ( V / V α ) is injective, Proposition 2.10 implies that ρ ( W S ) acts convex cocompactly on some properly convex open subset Ω of (cid:80) ( V ) (Defini-tion 2.3), with Λ orb Ω ( ρ ( W S )) = { [ x + ], [ x − ] } .Suppose now that t = A A =
4. By considering the Cartan matrix (cid:161) A A (cid:162) , wesee that u : = A v − v ∈ V v ∩ V α . We have assumed that (3.5) holds, hence α and α are linearly independent. Choose w ∈ Ker( α ) such that α ( w ) = u =
0, then V = span( v , w ) ⊕ V α ; the element ρ ( s s ) acts on V α trivially and on span( v , w ) as the matrix ( −
10 1 ).(iii) On the other hand, if u (cid:54)=
0, then V = span( u , v , w ) ⊕ V (cid:48) α where V (cid:48) α is any codimension-one subspace of V α not containing u ; the element ρ ( s s ) acts on V (cid:48) α trivially and onspan( u , v , w ) as the matrix (cid:181) A /2 00 1 −
10 0 1 (cid:182) .In both cases (ii) and (iii), the element ρ ( s s ) is unipotent and the Vinberg domain Ω Vin is the complement of the hyperplane V v + V α . Remark 3.11.
We shall say that ρ : W S → GL( V ) is symmetrizable if the Cartan matrix A is equivalent (Definition 3.3) to a symmetric matrix; in this case, ρ preserves a(possibly degenerate) symmetric bilinear form on V . Symmetrizable representationsof W S as a reflection group in V may be obtained by choosing arbitrary nonnegativenumbers ( λ i , j ) i < j , m i , j =∞ and setting A i , j = A j , i = − (2 + λ i , j ) for all i < j with m i , j = ∞ and A i , j = A j , i = − π / m i , j ) for all i < j with m i , j < ∞ . These representations canbe viewed as deformations of the Tits geometric representation, for which λ i , j = i < j with m i , j = ∞ .3.4. Types of Coxeter groups.
By [MV], every irreducible Coxeter group W S is either • spherical (i.e. finite), (i) 0 v K e r ( α ) v K e r ( α ) x + x − (cid:101) ∆ (cid:101) Ω Vin (ii) 0 v K e r ( α ) v K e r ( α ) (cid:101) ∆ (cid:101) Ω Vin (iii) (cid:80) K e r ( α ) (cid:80) K e r ( α ) [ v ] [ v ][ v ] [ v ][ u ] ∆ Ω Vin F IGURE
2. Infinite Coxeter groups on two generators as in Exam-ple 3.10, cases (i)–(ii)–(iii). The groups are shown acting on (cid:82) , (cid:82) and (cid:80) ( (cid:82) ) respectively. In the third panel, the points [ v ], [ v ] and [ u ] are atinfinity, and Ω Vin is the full affine chart. • affine (i.e. infinite and virtually abelian), • or large (i.e. there exists a surjective homomorphism of a finite-index subgroupof W S onto a nonabelian free group).3.5. Types of compatible matrices.
Let A ∈ M N ( (cid:82) ) be a matrix compatible with theirreducible Coxeter group W S (Definition 3.7). By construction, the matrix A (cid:48) : = − A ∈ M N ( (cid:82) ) has nonnegative entries. By the Perron–Frobenius theorem, A (cid:48) admits aneigenvector with positive entries corresponding to the highest eigenvalue of A (cid:48) ; sincethe Coxeter diagram of W S is connected, this vector is unique up to scale (see e.g. [Se,Th. 1.5]). It is also an eigenvector for the lowest eigenvalue of A . Definition 3.12.
The compatible matrix A is of negative (resp. zero , resp. positive ) type if the lowest eigenvalue of A is negative (resp. zero, resp. positive). Remark 3.13.
Suppose A = ( α i ( v j )) (cid:201) i , j (cid:201) N for some α = ( α , . . . , α N ) ∈ V ∗ N and v = ( v , . . . , v N ) ∈ V N . Let t = ( t , . . . , t N ) ∈ (cid:82) N be the Perron–Frobenius eigenvector of A (cid:48) = − A . Let x = (cid:80) Nj = t j v j ∈ V v . The i -th entry of A t is α i ( x ).(1) If A is of negative (resp. positive) type, then x (resp. − x ) belongs to the interior (cid:101) ∆ ◦ of the fundamental polyhedral cone (cid:101) ∆ of (3.4). In particular, (cid:101) ∆ ◦ is nonempty,i.e. (3.5) holds.(2) If A is of zero type, then x belongs to V v ∩ V α . Determining whether (3.5) holdsis more subtle: see [Vi1, Prop. 13]. Fact 3.14 ([Vi1, Th. 3]) . Let A ∈ M N ( (cid:82) ) be a matrix compatible with the Coxeter group W S ,and let τ = ( τ , . . . , τ N ) ∈ (cid:82) N .(1) If A is of negative type and both τ and A τ have all their entries (cid:202) , then τ = .(2) If A is of zero type and A τ has all its entries (cid:202) , then A τ = . ONVEX COCOMPACTNESS FOR COXETER GROUPS 19
Fact 3.15 ([Vi1, Prop. 22–23]) . Let W S be an irreducible Coxeter group and A ∈ M N ( (cid:82) ) a compatible Cartan matrix for W S (Definition 3.7). • The group W S is spherical if and only if A is of positive type; in this case A issymmetrizable. • Suppose W S is affine. – If A is of zero type, then A is symmetrizable and any representation ρ : W S → GL( V ) of W S as a reflection group (Definition 1.2) with Cartan ma-trix A is non-semisimple; if ρ ( W S ) is reduced, then it acts properly discon-tinuously and cocompactly on some affine chart of (cid:80) ( V ) , preserving someEuclidean metric. – If A is of negative type, then det A (cid:54)= and W S is of type (cid:101) A N − with N (cid:202) (see Appendix A); if furthermore N (cid:202) , then A is not symmetrizable. • If W S is large, then A is of negative type. Remark 3.16.
Let ρ : W S → GL( V ) be a representation of W S as a reflection group withCartan matrix A , and let ρ v , ρ α , and ρ α v be the induced representations of Remark 3.6.If A is of negative or positive type, then ρ v , ρ α , and ρ α v are representations of W S as areflection group, by Remark 3.13.(1); in particular, these representations are injective.If A is of zero type, then ρ v and ρ α v are never representations of W S as a reflectiongroup, as (3.5) fails by Fact 3.14.(2).3.6. Affine groups with Cartan matrices of negative type.
Let W S be a Coxetergroup of type (cid:101) A N − with N (cid:202) ρ : W S → GL( V ) a representationof W S as a reflection group in V , associated to some α = ( α , . . . , α N ) ∈ V ∗ N and v = ( v , . . . , v N ) ∈ V N . The case N = N (cid:202)
3. The Cartan matrix A = ( α i ( v j )) (cid:201) i , j (cid:201) N is then equivalent (Definition 3.3) to(3.8) − − a − − − a − − for some a >
0; it has determinant 2 − a − a − . The matrix A is of negative type if andonly if det A (cid:54)=
0, if and only if a (cid:54)=
1, if and only if A is not symmetrizable (Fact 3.14).The following is stated in [MV, Lem. 8] or [Ma2, Thm. 2.18]; we give a short proof forthe reader’s convenience. Lemma 3.17.
Let W S be a Coxeter group of type (cid:101) A N − with N (cid:202) and ρ : W S → GL( V ) arepresentation of W S as a reflection group in V , associated to some α = ( α , . . . , α N ) ∈ V ∗ N and v = ( v , . . . , v N ) ∈ V N . Suppose the Cartan matrix A = ( α i ( v j )) (cid:201) i , j (cid:201) N is of negativetype and ρ ( W S ) is reduced and dual-reduced (Definition 3.5). Then(1) dim( V ) = N;(2) ρ ( W S ) divides (i.e. acts properly discontinuously and cocompactly on) an opensimplex of (cid:80) ( V ) , namely Ω Vin ;(3) Ω Vin is the unique nonempty ρ ( W S ) -invariant convex open subset of (cid:80) ( V ) . Proof. (1) By Fact 3.15, we have det A (cid:54)=
0, hence the vectors v , . . . , v N ∈ V are linearlyindependent. Since ρ ( W S ) is dual-reduced, these vectors span V , hence dim V = N .(2) Without loss of generality, we may assume that A = ( α i ( v j )) (cid:201) i , j (cid:201) N is given by (3.8)and, since det A (cid:54)=
0, that in some appropriate basis of V (cid:39) (cid:82) N we have α ... α N = −
1. . . . . .. . . − − a − and v · · · v N = − a − − .The formula ρ ( s i ) = ( x (cid:55)→ x − α i ( x ) v i ) yields { ρ ( s i ) } (cid:201) i (cid:201) N = , . . . , , a a − .The first N − S N . We have(3.9) ρ (cid:161) ( s s . . . s N − )( s N − . . . s s ) s N (cid:162) = Diag( a − , 1, . . . , 1, a ),where Diag( u , . . . , u N ) ∈ M N ( (cid:82) ) denotes the diagonal matrix with entries u , . . . , u N ;from this it is easy to show that ρ ( W S ) (cid:39) (cid:101) A N − is isomorphic to S N (cid:110) (cid:90) N − . Here thenormal subgroup (cid:90) N − consists of all matrices of the form Diag( a ν , . . . , a ν N ) with ν i ∈ (cid:90) and ν +· · ·+ ν N =
0; we get a system of generators for this (cid:90) N − by permuting s , . . . , s N cyclically in (3.9).If a < a > (cid:101) ∆ = (cid:84) Ni = { α i (cid:201) } is the (cid:82) (cid:202) -span (resp. (cid:82) (cid:201) -span) of the column vectors ( a , . . . , a , 1, . . . , 1). In either case, ∆ is a closed (compact)projective simplex contained in the projectivized positive orthant (cid:80) (cid:161) (cid:82) N > (cid:162) . The Vinbergdomain Ω Vin , which is by definition (Section 3.3) the interior of the union of the ρ ( W S )-translates of ∆ , is equal to (cid:80) (cid:161) (cid:82) N > (cid:162) . Thus ρ ( W S ) divides Ω Vin .(3) The properly convex open cones of (cid:82) N preserved by the (cid:90) N − factor of ρ ( W S ) (cid:39) S N (cid:110) (cid:90) N − (acting as diagonal matrices) are precisely the orthants. The convex (notnecessarily properly convex) cones preserved by (cid:90) N − are the Cartesian products of N factors each equal to (cid:82) > , (cid:82) < , or (cid:82) . The only nontrivial such product which is S N -invariant after projectivization is (cid:80) (cid:161) (cid:82) N > (cid:162) . (cid:3) Strong irreducibility.
The following proposition will be used several times be-low. For reflection groups (Definition 1.2), the first two items were proved in [Vi1,Prop. 19] and [Vi1, Cor. to Prop. 19], and the third one in [Ma2, Th. 2.18].
Proposition 3.18.
Let W S be an infinite irreducible Coxeter group in N generatorsas in (3.1) and let ρ : W S → GL( V ) be a representation of W S generated by weaklycompatible reflections (Definition 3.2), associated to some α = ( α , . . . , α N ) ∈ V ∗ N andv = ( v , . . . , v N ) ∈ V N . Let V v and V α be as in Definition 3.5. Then(1) a linear subspace V (cid:48) of V is ρ ( W S ) -invariant if and only if V v ⊂ V (cid:48) or V (cid:48) ⊂ V α ; ONVEX COCOMPACTNESS FOR COXETER GROUPS 21 (2) the representation ρ is irreducible if and only if V v = V and V α = { } , i.e. if andonly if ρ ( W S ) is reduced and dual-reduced (Definition 3.5);(3) in this case, if W S large, then ρ is actually strongly irreducible. By strongly irreducible we mean that the restriction of ρ to any finite-index subgroupof W S is irreducible; equivalently, the group ρ ( W S ) does not preserve a finite union ofnontrivial subspaces of V . Proof. (1) If V v ⊂ V (cid:48) , then V (cid:48) is invariant under each ρ ( s i ) = ( x (cid:55)→ x − α i ( x ) v i ), henceunder ρ ( W S ); if V (cid:48) ⊂ V α , then V (cid:48) is pointwise fixed by each ρ ( s i ), hence by ρ ( W S ).Conversely, suppose V (cid:48) is ρ ( W S )-invariant. Let S (cid:48) be the set of generators s j of W S such that v j ∈ V (cid:48) , and S (cid:48)(cid:48) its complement in S . For any s i ∈ S (cid:48)(cid:48) and v (cid:48) ∈ V (cid:48) we have ρ ( s i )( v (cid:48) ) = v (cid:48) − α i ( v (cid:48) ) v i ∈ V (cid:48) , hence α i ( v (cid:48) ) =
0. In particular, for any s i ∈ S (cid:48)(cid:48) and s j ∈ S (cid:48) wehave α i ( v j ) =
0, hence m i , j = A . Since W S is irreducible, wehave S (cid:48) = S , in which case V (cid:48) ⊃ V v , or S (cid:48)(cid:48) = S , in which case V (cid:48) ⊂ V α .(2) Note that V v and V α are invariant subspaces of V with V v (cid:54)= { } and V α (cid:54)= V .Therefore, if ρ is irreducible, then V v = V and V α = { } . Conversely, if V v = V and V α = { } , then (1) implies that any invariant subspace is trivial, hence ρ is irreducible.(3) Suppose by contradiction that W S is large and ρ is irreducible but not stronglyirreducible: this means that there is a finite collection F of nontrivial subspaces of V which is preserved by the action of ρ . We may assume that F is closed under intersec-tion (excluding the trivial subspace).We first claim that F does not contain any one-dimensional subspace U . Indeed,if U ∈ F is one-dimensional, then for any nonzero u ∈ U , the set ρ ( W S ) · u must span V since ρ is irreducible, hence this set contains a basis B of V . Consider the setof elements of ρ ( W S ) whose action preserves each individual subspace of F ; it is afinite-index subgroup H of ρ ( W S ). The basis B is a simultaneous eigenbasis for allelements of H , hence H is abelian, and so W S is virtually abelian. This contradicts theassumption that W S is a large irreducible Coxeter group.Next, let U be a subspace in F of minimal dimension. We have dim U (cid:202) i , the subspace ρ ( s i ) · U also belongs to F . Since ρ ( s i ) isa reflection, if we had ρ ( s i ) · U (cid:54)= U , then ρ ( s i ) · U ∩ U would be a subspace in F ofdimension dim U −
1, nontrivial since dim U (cid:202)
2, contradicting the minimality of U .Thus ρ ( s i ) · U = U for all i , and so ρ ( W S ) · U = U , contradicting the irreducibility of ρ . (cid:3) The following consequence of Proposition 3.18 will be used in Sections 4.2 and 6.2.
Corollary 3.19.
Let W S be an irreducible Coxeter group, and ρ : W S → GL( V ) a rep-resentation of W S as a reflection group in V , associated to some α = ( α , . . . , α N ) ∈ V ∗ N and v = ( v , . . . , v N ) ∈ V N , such that the Cartan matrix A = ( α i ( v j )) (cid:201) i , j (cid:201) N is of negativetype. Then(1) the proximal limit set Λ ρ ( W S ) of ρ ( W S ) is nonempty and contained in (cid:80) ( V v ) ; (2) for any nonempty ρ ( W S ) -invariant closed properly convex subset C of Ω Vin , wehave C ∩ (cid:80) ( V v ) (cid:54)= (cid:59) .Proof. (1) Let ρ α v : W S → GL( V α v ) be the representation induced by ρ , as in Remark 3.6.It is easy to check that ρ ( W S ) contains a proximal element in (cid:80) ( V ) if and only if ρ α v ( W S )contains a proximal element in (cid:80) ( V α v ) (see e.g. (3.10) below). This last statement is trueby Proposition 2.6 since ρ α v ( W S ) acts irreducibly on (cid:80) ( V α v ) by Proposition 3.18.(2).By Remark 3.13.(1), there exists a point y ∈ Ω Vin ∩ (cid:80) ( V v ). Since V v is ρ ( W S )-invariant,we have ρ ( W S ) · y ⊂ (cid:80) ( V v ), and so the proximal limit set Λ ρ ( W S ) is contained in (cid:80) ( V v ) byProposition 2.5.(1).(2) By Proposition 2.5.(1) and (1) above, we have Λ ρ ( W S ) ⊂ C ∩ (cid:80) ( V v ) where C is theclosure of C in (cid:80) ( V ). Therefore the convex hull of Λ ρ ( W S ) in Ω Vin is contained in C ∩ (cid:80) ( V v ). This convex hull is nonempty because Ω Vin ∩ (cid:80) ( V v ) (cid:54)= (cid:59) and the projective spanof Λ ρ ( W S ) is the whole of (cid:80) ( V v ), by Proposition 3.18.(1). (cid:3) Block triangular decomposition.
Let ρ : W S → GL( V ) be a representation of W S generated by weakly compatible reflections (Definition 3.2), associated to some α ∈ V ∗ N and v ∈ V N , with Cartan matrix A . Choose a complementary subspace U (cid:48) of V α ∩ V v in V v , a complementary subspace U (cid:48)(cid:48) of V α ∩ V v in V α , and a complementarysubspace U (cid:48)(cid:48)(cid:48) of V α + V v in V . Then V = (cid:124) (cid:123)(cid:122) (cid:125) V α U (cid:48)(cid:48) ⊕ V v (cid:122) (cid:125)(cid:124) (cid:123) ( V α ∩ V v ) ⊕ U (cid:48) ⊕ U (cid:48)(cid:48)(cid:48) .By the definition (3.3) of the ρ ( s i ), in a basis adapted to this decomposition of V , theelements of ρ ( W S ) are matrices of the form(3.10) ρ ( γ ) = Id 0 0 00 Id ∗ ∗ ρ α v ( γ ) ∗ ρ v ( γ ) ρ α ( γ )where ρ v , ρ α , ρ α v are the induced representations from Remark 3.6, and we implicitlyidentify V α with U (cid:48) ⊕ U (cid:48)(cid:48)(cid:48) and V α v with U (cid:48) . (The zeros in (3.10) come from the definitionof V α in columns 1 and 2, and of V v in rows 1 and 4.) Further, dim( U (cid:48) ) = rank( A ) by[Vi1, Prop. 15] and ρ α v is irreducible by Proposition 3.18.(2). Lemma 3.20.
The representation ρ is semisimple if and only if V = V α ⊕ V v , i.e. ( U (cid:48)(cid:48) , V α ∩ V v , U (cid:48) , U (cid:48)(cid:48)(cid:48) ) = ( V α , { } , V v , { } ) , in which case ρ has the form (cid:181) Id 00 ρ α v (cid:182) .Proof. If ρ is semisimple, then V α (resp. V v ) admits a ρ ( W S )-invariant complementarysubspace in V , which must contain V v (resp. be contained in V α ) by Proposition 3.18.(1);we deduce V = V α ⊕ V v . Conversely, if V = V α ⊕ V v , then ρ has the given matrix form,hence ρ is semisimple since ρ α v is irreducible. (cid:3) ONVEX COCOMPACTNESS FOR COXETER GROUPS 23
Parametrizing characters by equivalence classes of Cartan matrices.
Sup-pose W S is an infinite irreducible Coxeter group. Let A be an N × N real matrix whichis weakly compatible with W S (Definition 3.1).If rank( A ) = dim( V ), then we may directly construct α = ( α , . . . , α N ) ∈ V ∗ N and v = ( v , . . . , v N ) ∈ V N with ( α i ( v j )) Ni , j = = A as follows. Choose rank( A ) linearly inde-pendent rows of A giving a basis of the row space of A ; the subset of α correspondingto those rows must form a basis B ∗ of V ∗ . Each remaining row is a linear combinationof the chosen row basis; the corresponding element of α must be the corresponding lin-ear combination of B ∗ . This uniquely determines α , up to the choice of the basis B ∗ .The vectors of v are then determined by their coordinates with respect to B ∗ whichare the entries of A . Hence, up to the action of GL( V ), the Cartan matrix A uniquelydetermines α and v . The corresponding representation ρ : W S → GL( V ) generated byweakly compatible reflections is uniquely determined, up to conjugacy, by the equiva-lence class of A ; it is irreducible by Proposition 3.18.(2).If rank( A ) < dim( V ), then splitting V into a trivial summand plus a subspace ofdimension rank( A ) and applying the same process yields a unique conjugacy class ofsemisimple representations. Hence we get the following fact. Fact 3.21.
For any N × N real matrix A of rank (cid:201) dim( V ) which is weakly compatiblewith W S (Definition 3.1), there is a unique conjugacy class of semisimple representations ρ : W S → GL( V ) with Cartan matrix A . Further, the constructive correspondence described above implies the following para-metrization statement, which will be used in Section 8.
Fact 3.22.
The map assigning to a conjugacy class of semisimple representations theequivalence class of its Cartan matrices is a homeomorphism from the open subset of χ ( W S , GL( V )) consisting of representations generated by weakly compatible reflectionsto the space of equivalence classes of N × N matrices of rank (cid:201) dim( V ) that are weaklycompatible with W S . Note that if rank( A ) < dim( V ), then there are also non-semisimple conjugacy classesassociated to A . Invariants for these are more subtle to describe, see [Vi1, Prop. 15].4. M AXIMALITY OF THE V INBERG DOMAIN
In this section we establish the following; see Definition 3.5 for the notions of reducedand dual-reduced.
Proposition 4.1.
Let W S be an irreducible Coxeter group in N (cid:202) generators and ρ : W S → GL( V ) a representation of W S as a reflection group in V (Definition 1.2) with aCartan matrix A = ( α i ( v j )) (cid:201) i , j (cid:201) N of negative type (Definition 3.12). Then(1) if ρ ( W S ) is reduced, then Ω Vin is properly convex and it is maximal for inclusionamong all nonempty ρ ( W S ) -invariant convex open subsets of (cid:80) ( V ) ; Ω (cid:48) Ω Vin ∆ (cid:91) z y (cid:48)(cid:48) y (cid:48) y F IGURE
3. Illustration for the proof of Proposition 4.1.(1) (2) if ρ ( W S ) is reduced and dual-reduced and N (cid:202) , then Ω Vin is the unique maxi-mal nonempty ρ ( W S ) -invariant convex open subset of (cid:80) ( V ) ;(3) in general, if N (cid:202) , then any ρ ( W S ) -invariant properly convex open subset of (cid:80) ( V ) is contained in the Vinberg domain Ω Vin . Remark 4.2.
In the setting of Proposition 4.1, if N = ρ ( W S ) is reduced and dual-reduced (see Example 3.10.(i)), then (cid:80) ( V ) (cid:224) Ω Vin is also a nonempty ρ ( W S )-invariantproperly convex open subset of (cid:80) ( V ).If ρ is defined by α = ( α , . . . , α N ) ∈ V ∗ N and v = ( v , . . . , v N ) ∈ V N , we denote by V v the span of v , . . . , v N and by V α the intersection of the kernels of α , . . . , α N , as inDefinition 3.5.4.1. The reduced case.
We first prove (1) and (2).
Proof of Proposition 4.1. (1) . Suppose ρ ( W S ) is reduced (i.e. V α = { } ), and let Ω (cid:48) be a ρ ( W S )-invariant convex open subset of (cid:80) ( V ) containing Ω Vin .We first check that Ω (cid:48) is properly convex. Suppose by contradiction that it is not.The largest linear subspace V (cid:48) of V contained in the closure of a convex lift of Ω (cid:48) to V defines a nontrivial invariant projective subspace (cid:80) ( V (cid:48) ) which is disjoint from Ω (cid:48) (and from any affine chart containing Ω (cid:48) ). Since ρ ( W S ) is reduced, Proposition 3.18.(1)implies that V v ⊂ V (cid:48) , hence (cid:80) ( V v ) ⊂ (cid:80) ( V ) (cid:224) Ω (cid:48) ⊂ (cid:80) ( V ) (cid:224) Ω Vin . This contradicts the fact (cid:80) ( V v ) ∩ Ω Vin (cid:54)= (cid:59) by Remark 3.13. Thus Ω (cid:48) must be properly convex.We now check that Ω (cid:48) = Ω Vin . Suppose by contradiction that there is a point y ∈ Ω (cid:48) (cid:224) Ω Vin ; Figure 3 illustrates the proof. Let z lie in the interior of the fundamentaldomain ∆ (cid:91) = ∆ ∩ Ω Vin for the action of ρ ( W S ) on Ω Vin . Recall (Fact 3.8) that ∆ (cid:91) is equal ONVEX COCOMPACTNESS FOR COXETER GROUPS 25 to ∆ minus the union of all faces with infinite stabilizer. The quotient ρ ( W S ) \ Ω Vin isan orbifold which is homeomorphic to ∆ (cid:91) , with mirrors on the reflection walls. Theintersection of the segment [ z , y ] with Ω Vin is a half-open segment [ z , y (cid:48) ) where y (cid:48) ∈ ∂ Ω Vin ∩ Ω (cid:48) . It has finite length in the Hilbert metric d Ω (cid:48) (see Section 2.1). The imageof [ z , y (cid:48) ) in the quotient may be viewed as a billiard path q : [ z , y (cid:48) ) → ∆ (cid:91) (where thereflection associated to a mirror wall is used to determine how the trajectory reflectsoff that wall). Any point of ∆ (cid:91) is the center of a d Ω (cid:48) -ball contained in Ω Vin , hence disjointfrom ρ ( W S ) · y (cid:48) ; it follows that q ( t ) can only accumulate on points of ∆ (cid:224) ∆ (cid:91) as t → y (cid:48) .Any such accumulation point y (cid:48)(cid:48) belongs to a face of ∆ with infinite stabilizer, hence y (cid:48)(cid:48) ∈ ∂ Ω (cid:48) since ρ ( W S ) acts properly discontinuously on Ω (cid:48) . It follows that d Ω (cid:48) ( z , q ( t ))goes to infinity as t → y (cid:48) . By the triangle inequality, we have d Ω (cid:48) ( z , y (cid:48) ) > d Ω (cid:48) ( z , q ( t )),contradicting the fact that d Ω (cid:48) ( z , y (cid:48) ) is finite. (cid:3) Proof of Proposition 4.1. (2) . Suppose ρ ( W S ) is reduced and dual-reduced and N (cid:202)
3. ByProposition 3.18.(2), the representation ρ is irreducible, hence any ρ ( W S )-invariantconvex open subset Ω of (cid:80) ( V ) has to be properly convex. Indeed, the largest linearsubspace of V contained in the closure of a convex lift of Ω to V is ρ ( W S )-invariant.Suppose W S is affine. Since ρ is irreducible (hence semisimple), Fact 3.15 impliesthat W S is of type (cid:101) A N − with N = dim( V ). Since N (cid:202)
3, Lemma 3.17 applies: the set Ω Vin is the unique ρ ( W S )-invariant convex open subset of (cid:80) ( V ).Suppose W S is large. By Proposition 3.18.(3), the representation ρ is strongly irre-ducible. By Proposition 2.6, there is a unique maximal ρ ( W S )-invariant properly convexopen set Ω max containing all other invariant properly convex open sets; this must bethe Vinberg domain Ω max = Ω Vin by Proposition 4.1.(1). (cid:3)
Remark 4.3.
Suppose we are in the setting of Proposition 4.1, namely W S is irreducibleand the Cartan matrix A is of negative type.(1) If ρ ( W S ) is reduced, then Ω Vin is properly convex by Proposition 4.1.(1). Onthe other hand, if ρ ( W S ) is not reduced, then Ω Vin is not properly convex, for itcontains the projective subspace (cid:80) ( V α ) in its boundary.(2) If ρ ( W S ) is not dual-reduced, then there is a ρ ( W S )-invariant convex open subsetof (cid:80) ( V ) which is not contained in Ω Vin , namely the complement U in (cid:80) ( V ) ofany projective hyperplane containing (cid:80) ( V v ). Indeed, U is ρ ( W S )-invariant byProposition 3.18.(1), and U (cid:54)⊂ Ω Vin because U is an affine chart, Ω Vin is convex,and Ω Vin ∩ (cid:80) ( V v ) (cid:54)= (cid:59) by Remark 3.13. See Figure 4.4.2. The general case.
For any representation ρ (cid:48) : W S → GL( V (cid:48) ) of our Coxeter group W S , we write Λ ρ (cid:48) for the proximal limit set Λ ρ (cid:48) ( W S ) of ρ (cid:48) ( W S ) in (cid:80) ( V (cid:48) ), as in Defini-tion 2.2, and Λ ∗ ρ (cid:48) : = Λ ρ (cid:48)∗ ( W S ) for the proximal limit set of ρ (cid:48)∗ ( W S ) in (cid:80) ( V (cid:48)∗ ), where ρ (cid:48)∗ : W S → GL( V (cid:48)∗ ) is the dual representation. Proof of Proposition 4.1. (3) . By Corollary 3.19.(1), there exists γ ∈ W S such that ρ ( γ )is proximal in (cid:80) ( V ); then ρ ( γ − ) is proximal in (cid:80) ( V ∗ ), and so Λ ∗ ρ (cid:54)= (cid:59) . By Proposi-tion 2.5.(3), it is sufficient to prove the following two claims: ∆∆ (cid:80) ( V v ) ∆ ∩ (cid:80) ( V v ) F IGURE
4. Illustration of Remark 4.3.(2) for N =
4, where V v is ahyperplane in V = (cid:82) and the affine reflections preserving the chart (cid:80) ( V ) (cid:224) (cid:80) ( V v ) (cid:39) (cid:82) are chosen with linear parts in O(2, 1). As above, ∆ isthe fundamental polytope for ρ , and Ω Vin is the interior of (cid:83) γ ρ ( γ ) · ∆ . Here Ω Vin intersects the chart in two connected components, each of which isa domain of dependence as in [Me] (see also [DGK2, Ex. 11.12]).(i) the Vinberg domain Ω Vin is a full connected component of U : = (cid:80) ( V ) (cid:224) (cid:91) [ ϕ ] ∈ Λ ∗ ρ (cid:80) (Ker( ϕ ));(ii) the set U admits only one ρ ( W S )-invariant connected component.We first prove (i). Consider V α = V / V α and the representation ρ α : W S → GL( V α )induced by ρ , as in Remark 3.6: since A is assumed of negative type, ρ α is a represen-tation of W S as a reflection group (Remark 3.16). By Propositions 2.5.(3) and 4.1.(1),the corresponding Vinberg domain Ω α Vin ⊂ (cid:80) ( V α ) is a full connected component of(4.1) U α : = (cid:80) ( V α ) (cid:224) (cid:91) [ ϕ ] ∈ Λ ∗ ρα (cid:80) (Ker( ϕ )).Let π : (cid:80) ( V ) (cid:224) (cid:80) ( V α ) → (cid:80) ( V α ) be the natural projection. On the one hand, the funda-mental reflection polytope ∆ α ⊂ (cid:80) ( V α ) for ρ α ( W S ) satisfies π − ( ∆ α ) = ∆ (cid:224) (cid:80) ( V α ), hence Ω Vin = π − ( Ω α Vin ). On the other hand, the connected components of U are exactly thepreimages under π of the connected components of U α . Indeed, the dual ( V α ) ∗ of V α identifies with the annihilator ( V ∗ ) α ⊂ V ∗ of V α , i.e. the set of linear forms ϕ ∈ V ∗ that vanish on V α , which is also the span of the α i in V ∗ . The dual action of ρ ( W S )on V ∗ exchanges the roles of the v i and the α i : namely, ρ ( s i ) acts on V ∗ as a reflec-tion in the hyperplane defined by v i , with ( − α i . Therefore the proximallimit set Λ ∗ ρ of ρ ∗ ( W S ) in (cid:80) ( V ∗ ) is contained in the subspace (cid:80) (( V ∗ ) α ) = (cid:80) (( V α ) ∗ ), andequal to the proximal limit set Λ ∗ ρ α of ( ρ α ) ∗ ( W S ) in that subspace. This implies thatthe connected components of U are exactly the preimages under π of the connectedcomponents of U α , and completes the proof of (i).We now prove (ii), assuming N (cid:202)
3. By the proof of (i), it is sufficient to check that theset U α of (4.1) has a unique ρ ( W S )-invariant connected component, for the projection π : ONVEX COCOMPACTNESS FOR COXETER GROUPS 27 (cid:80) ( V ) (cid:224) (cid:80) ( V α ) → (cid:80) ( V α ) is ρ ( W S )-equivariant. Assume by contradiction that there existsa ρ ( W S )-invariant connected component Ω (cid:48) of U α different from the Vinberg domain Ω α Vin . Intersecting Ω α Vin with (cid:80) ( V α v ) yields the Vinberg domain for ρ α v ( W S ) in (cid:80) ( V α v ).Since ρ α v ( W S ) is both reduced and dual reduced, Ω α Vin ∩ (cid:80) ( V α v ) is the unique maximalnonempty ρ α v ( W S )-invariant convex open subset of (cid:80) ( V α v ) by Proposition 4.1.(2). Now,it may be that Ω (cid:48) does not intersect (cid:80) ( V α v ). However, Ω (cid:48) contains the proximal limitset Λ ρ α , hence Ω (cid:48) ∩ (cid:80) ( V α v ) is a closed ρ α v ( W S )-invariant convex subset of (cid:80) ( V α v ) whoseinterior, necessarily nonempty by Proposition 3.18, is contained in Ω α Vin ∩ (cid:80) ( V α v ). Since Ω α Vin is open, we find that Ω α Vin and Ω (cid:48) must overlap, hence they are equal. (cid:3)
5. T
HE MINIMAL INVARIANT CONVEX SUBSET OF THE V INBERG DOMAIN
In this section we consider an irreducible Coxeter group W S and a representation ρ : W S → GL( V ) of W S as a reflection group, associated to some α = ( α , . . . , α N ) ∈ V ∗ N and v = ( v , . . . , v N ) ∈ V N , such that the Cartan matrix A = ( α i ( v j )) (cid:201) i , j (cid:201) N is of neg-ative type (Definition 3.12) and ρ ( W S ) is reduced and dual-reduced (Definition 3.5).By Proposition 3.18, such a representation ρ is irreducible. By Proposition 4.1, theVinberg domain Ω Vin is properly convex and contains all other ρ ( W S )-invariant openconvex subsets of (cid:80) ( V ). By Proposition 2.6, there is a unique smallest nonempty Γ -invariant convex open subset Ω min of Ω Vin . The goal of this section is to describe Ω min (Lemma 5.1 and Theorem 5.2).5.1. Reflections in the dual projective space.
Recall that the dual action ρ ∗ of W S on V ∗ exchanges the roles of the v i and the α i : namely, ρ ∗ ( s i ) is a reflection in thehyperplane defined by v i , with ( − α i . The Cartan matrix for ρ ∗ ( W S ) isthe transpose of the Cartan matrix A = ( α i ( v j )) (cid:201) i , j (cid:201) N for ρ ( W S ). Similarly to Sec-tion 3.3, we define a closed fundamental polyhedral cone (cid:101) D ⊂ V ∗ for ρ ∗ ( W S ), cut out bythe kernels of v , . . . , v N seen as linear forms on V ∗ . The group ρ ∗ ( W S ) acts properlydiscontinuously on a nonempty convex open cone (cid:101) O Vin of V ∗ , namely the interior ofthe union of all ρ ∗ ( W S )-translates of (cid:101) D . The image O Vin ⊂ (cid:80) ( V ∗ ) of (cid:101) O Vin is the Vinbergdomain of ρ ∗ ; by Proposition 4.1, it is again properly convex and contains all other ρ ∗ ( W S )-invariant convex subsets of (cid:80) ( V ∗ ). Since duality between properly convex setsreverses inclusion, the minimal ρ ( W S )-invariant properly convex subset Ω min of (cid:80) ( V )is the dual of the maximal ρ ∗ ( W S )-invariant properly convex subset O Vin of (cid:80) ( V ∗ ), i.e. Ω min = O ∗ Vin .We define a dualization operation ∗ for closed sets: if F is a closed convex cone in V ,then F ∗ is the closed convex cone in V ∗ defined by F ∗ : = (cid:169) ϕ ∈ V ∗ | ϕ ( x ) (cid:201) ∀ x ∈ F (cid:170) .Note that F ∗∗ = F . For (cid:101) D , (cid:101) O Vin ⊂ V ∗ as above, we have(5.1) (cid:101) D ∗ = N (cid:88) j = (cid:82) (cid:202) v j ⊂ V and (cid:101) O Vin ∗ ⊂ V is a lift (cid:101) Ω min of Ω min . Lemma 5.1.
If W S is an irreducible Coxeter group and ρ : W S → GL( V ) a representationof W S as a reflection group with a Cartan matrix of negative type, such that ρ ( W S ) isreduced and dual-reduced, then Ω min = (cid:92) γ ∈ W S ρ ( γ ) · D ∗ , where D ∗ ⊂ (cid:80) ( V ) is the image of (cid:101) D ∗ .Proof. It is a well-known fact (see e.g. [Sa, Th. 2]) that if { (cid:101) C i } i ∈ I is a family of closedconvex cones of V , then ( (cid:84) i ∈ I (cid:101) C i ) ∗ is the closed convex hull Conv ( (cid:83) i ∈ I (cid:101) C ∗ i ) of the unionof their duals. This implies (cid:101) Ω min ∗ = (cid:101) O Vin = Conv (cid:179) (cid:91) γ ∈ W S ρ ( γ ) · (cid:101) D (cid:180) = (cid:179) (cid:92) γ ∈ W S ρ ( γ ) · (cid:101) D ∗ (cid:180) ∗ ,where for the second equality we use the fact that (cid:101) O Vin = (cid:83) γ ∈ W S ρ ( γ ) · (cid:101) D is convex. Wethen apply F ∗∗ = F to the closed convex cones F = (cid:101) Ω min and F = (cid:84) γ ∈ W S ρ ( γ ) · (cid:101) D ∗ . (cid:3) Pruning the fundamental polytope.
Consider the following subsets of the fun-damental polytope (cid:101) ∆ :(5.2) (cid:101) Σ : = (cid:101) ∆ ∩ (cid:101) D ∗ = (cid:189) x = N (cid:88) j = t j v j ∈ V (cid:175)(cid:175)(cid:175) α i ( x ) (cid:201) t j (cid:202) ∀ (cid:201) i , j (cid:201) N (cid:190) ⊂ (cid:101) Ω Vin and(5.3) (cid:101) Σ (cid:91) : = (cid:101) ∆ ∩ Int( (cid:101) D ∗ ) = (cid:189) x = N (cid:88) j = t j v j ∈ V (cid:175)(cid:175)(cid:175) α i ( x ) (cid:201) t j > ∀ (cid:201) i , j (cid:201) N (cid:190) ⊂ (cid:101) Σ .Note that (cid:101) Σ (cid:91) (hence (cid:101) Σ ) has nonempty interior by Remark 3.13. Let Σ and Σ (cid:91) be therespective images of (cid:101) Σ and (cid:101) Σ (cid:91) in (cid:80) ( V ). See Figures 5 and 6 for an illustration in someparticular cases. Here is the main result of this section. Theorem 5.2.
If W S is an irreducible Coxeter group and ρ : W S → GL( V ) a representa-tion of W S as a reflection group with a Cartan matrix A = ( A i , j ) (cid:201) i , j (cid:201) N of negative type,such that ρ ( W S ) is reduced and dual-reduced, then Ω min = (cid:91) γ ∈ W S ρ ( γ ) · Σ (cid:91) and Σ (cid:91) is a fundamental domain for the action of ρ ( W S ) on Ω min . Theorem 5.2 is a consequence of Lemma 5.1 and of the following two lemmas.
Lemma 5.3.
In the setting of Theorem 5.2, the set Σ (cid:91) is contained in the set ∆ (cid:91) of Fact 3.8and (cid:83) γ ∈ W S ρ ( γ ) · Σ (cid:91) is the interior of (cid:83) γ ∈ W S ρ ( γ ) · Σ . Lemma 5.4.
In the setting of Theorem 5.2, we have (cid:83) γ ∈ W S ρ ( γ ) · Σ ⊂ (cid:84) γ ∈ W S ρ ( γ ) · D ∗ . ONVEX COCOMPACTNESS FOR COXETER GROUPS 29 t = t = Σ Ω
Vin ∆ Ker( α ) K e r ( α ) K e r ( α ) F IGURE
5. The sets ∆ , Σ , and Ω Vin for W S : = 〈 σ , σ , σ | σ i = = ( σ σ ) 〉 acting on V = (cid:82) as a reflection group, preserving a copy of (cid:72) in (cid:80) ( V ).Here ( t , t , t ) (cid:55)→ t v + t v + t v gives coordinates on V . The set ∆ (light gray) is a triangle bounded by the hyperplanes Ker( α i ), and Σ (darkgray) is the truncation of ∆ by the hyperplanes { t i = } (note that { t = } is at infinity). The set Ω Vin is the interior of the W S -orbit of ∆ (hereapproximated by 8 iterates), and contains Σ .F IGURE
6. Pictures of the sets ∆ , Σ and approximate pictures of the sets Ω Vin , C Σ and both Ω Vin and C Σ together for a reflection group represen-tation of a right-angled hexagon group in GL(4, (cid:82) ). Proof of Theorem 5.2 assuming Lemmas 5.3 and 5.4.
It follows from Lemma 5.1 that Ω min ∩ ∆ ⊂ D ∗ ∩ ∆ = Σ , hence Ω min ∩ (cid:83) γ ∈ W S ρ ( γ ) · ∆ ⊂ (cid:83) γ ∈ W S ρ ( γ ) · Σ , and so Ω min ∩ Ω Vin ⊂ (cid:83) γ ∈ W S ρ ( γ ) · Σ . Taking interiors and applying Lemma 5.3, we find Ω min ⊂ (cid:83) γ ∈ W S ρ ( γ ) · Σ (cid:91) .By Lemmas 5.1 and 5.4, we have (cid:83) γ ∈ W S ρ ( γ ) · Σ ⊂ Ω min . By taking interiors andapplying Lemma 5.3, we obtain (cid:83) γ ∈ W S ρ ( γ ) · Σ (cid:91) ⊂ Ω min . (cid:3) Proof of Lemma 5.3.
By Fact 3.8, the set (cid:101) ∆ (cid:91) is equal to (cid:101) ∆ minus its faces ofinfinite stabilizer. Let us check that any point of Σ (cid:91) = ∆ ∩ Int( D ∗ ) has finite stabilizerin W S ; equivalently, we can work in the vector space V and check that any point of (cid:101) Σ (cid:91) has finite stabilizer in W S . Suppose by contradiction that x ∈ (cid:101) Σ (cid:91) has infinite stabilizer.This stabilizer is the standard subgroup W S x where S x = { s i ∈ S | α i ( x ) = } . Let W T bean irreducible standard subgroup of W S x which is still infinite. The Cartan submatrix A W T : = ( A i , j ) s i , s j ∈ T is of negative or zero type. Since x ∈ (cid:101) Σ (cid:91) = (cid:101) ∆ ∩ Int( (cid:101) D ∗ ), we can write x = (cid:80) Nj = t j v j where α i ( x ) (cid:201) t j > (cid:201) i , j (cid:201) N . For any s i ∈ T , we have0 = α i ( x ) = (cid:88) { j | s j ∈ T } A i , j t j + (cid:88) { j | s j ∉ T } A i , j t j .(5.4)The terms of the second sum are all nonpositive, hence the first sum is nonnegative.By Fact 3.14.(1), the Cartan submatrix A W T cannot be of negative type, hence it is ofzero type. By Fact 3.14.(2), the first sum of (5.4) must then be zero. This implies thatthe second sum is zero, and so every term of the second sum is zero. However, since W S is irreducible, there exist s i ∈ T and s j ∈ S (cid:224) T such that A i , j <
0. Then A i , j t j = t j =
0, contradicting the fact that t j >
0. This completes the proof that Σ (cid:91) ⊂ ∆ (cid:91) .Let us now check that (cid:83) γ ∈ W S ρ ( γ ) · Σ (cid:91) is the interior of (cid:83) γ ∈ W S ρ ( γ ) · Σ . Observe that forany x ∈ ∆ (cid:91) , an open neighborhood of x in Ω Vin is given by (cid:83) γ ∈ W Sx ρ ( γ ) · U where U is arelatively open neighborhood of x in ∆ (cid:91) which touches only the reflection walls contain-ing x , and ρ ( W S x ) is the (finite) subgroup of ρ ( W S ) generated by the reflections alongwalls containing x . It follows that for any subset R of ∆ (cid:91) , the interior of (cid:83) γ ∈ W S ρ ( γ ) · R is (cid:83) γ ∈ W S ρ ( γ ) · R (cid:48) where R (cid:48) is the relative interior of R in ∆ (cid:91) . Since Σ (cid:91) is the relative inte-rior of Σ ∩ ∆ (cid:91) in ∆ (cid:91) , we obtain that (cid:83) γ ∈ W S ρ ( γ ) · Σ (cid:91) is the interior of (cid:161) (cid:83) γ ∈ W S ρ ( γ ) · Σ (cid:162) ∩ Ω Vin ,hence of (cid:83) γ ∈ W S ρ ( γ ) · Σ .5.4. Proof of Lemma 5.4.
It is enough to establish that(5.5) (cid:101) ∆ = (cid:169) x ∈ V | ρ ( γ ) x − x ∈ (cid:101) D ∗ ∀ γ ∈ W S (cid:170) .Indeed, for any x ∈ (cid:101) Σ = (cid:101) ∆ ∩ (cid:101) D ∗ and any γ ∈ W S we then have ρ ( γ ) x ∈ (cid:101) D ∗ + x ⊂ (cid:101) D ∗ , hence (cid:83) γ ∈ W S ρ ( γ ) · Σ ⊂ D ∗ , and so (cid:83) γ ∈ W S ρ ( γ ) · Σ ⊂ (cid:84) γ ∈ W S D ∗ .We now establish (5.5), following [Kac, Prop. 3.12].We say that γ = s i · · · s i m is a reduced expression for γ ∈ W S if m is minimal amongall possible expressions of γ as a product of the s i . The integer m is called the length of γ and is denoted by (cid:96) ( γ ). Since all the relations of W S have even length, we have | (cid:96) ( γ s i ) − (cid:96) ( γ ) | = γ ∈ W S and s i ∈ S . Lemma 5.5 (see [Kac, Lem. 3.11]) . Let W S be a Coxeter group and ρ : W S → GL( V ) arepresentation of W S as a reflection group in V , associated to some α = ( α , . . . , α N ) ∈ V ∗ N and v = ( v , . . . , v N ) ∈ V N . For any γ ∈ W S and s i ∈ S, the following are equivalent:(1) (cid:96) ( γ s i ) > (cid:96) ( γ ) ;(2) ρ ∗ ( γ ) α i belongs to the nonnegative span (cid:101) ∆ ∗ of α , . . . , α N in V ∗ ;(3) ρ ( γ ) v i belongs to the nonnegative span (cid:101) D ∗ of v , . . . , v N in V . ONVEX COCOMPACTNESS FOR COXETER GROUPS 31
In particular, if γ = s i · · · s i m is a reduced expression for γ ∈ W S , then ρ ( s i · · · s i m − )( v i m ) ∈ (cid:101) D ∗ . Proof of Lemma 5.5.
We give a proof of (1) ⇔ (2); the proof of (1) ⇔ (3) is similar usingdual objects. Recall that (cid:101) ∆ = { x ∈ V | α i ( x ) (cid:201) ∀ (cid:201) i (cid:201) N } . The linear form α ∈ V ∗ lies in (cid:101) ∆ ∗ if and only if α ( (cid:101) ∆ ) (cid:201)
0. Hence, ρ ∗ ( γ ) α i ∈ (cid:101) ∆ ∗ if and only if α i ( ρ ( γ − ) (cid:101) ∆ ) (cid:201) s i ∈ S and γ ∈ W S , the cones (cid:101) ∆ ◦ and ρ ( γ ) (cid:101) ∆ ◦ lieon opposite sides of the hyperplane Ker( α i ) ∩ (cid:101) Ω Vin of (cid:101) Ω Vin fixed by ρ ( s i ) if and only if (cid:96) ( γ ) > (cid:96) ( s i γ ). Hence, α i ( ρ ( γ − ) (cid:101) ∆ ) (cid:201) (cid:96) ( s i γ − ) > (cid:96) ( γ − ), completing theproof. (cid:3) Proof of (5.5) . The inclusion ⊃ is obvious, by (5.1) and the definitions (3.3) of ρ ( s i ) and(3.4) of (cid:101) ∆ . We prove the reverse inclusion by induction on m = (cid:96) ( γ ). For m =
1, it is thedefinition of (cid:101) ∆ . If m >
1, let γ = s i · · · s i m . We have ρ ( γ ) x − x = (cid:161) ρ ( s i · · · s i m − ) x − x (cid:162) + ρ ( s i · · · s i m − ) (cid:161) ρ ( s i m ) x − x (cid:162) and we apply the inductive assumption to the first summand and Lemma 5.5 to thesecond summand, using the equality ρ ( s i m ) x − x = − α i m ( x ) v i m from (3.3). (cid:3) The fundamental domain Σ ∩ ∆ (cid:91) of the closed convex core. As in Section 5.3,the stabilizer of x in W S is the standard subgroup W S x where S x : = (cid:169) s i ∈ S | α i ( x ) = (cid:170) .Given a decomposition x = (cid:80) Nj = t j v j ∈ V such that t : = ( t j ) Nj = ∈ (cid:82) N (cid:202) , we introduce S + , t : = (cid:169) s j ∈ S | t j > (cid:170) , S tx : = { s j ∈ S x | t j = } , and S + , tx : = { s j ∈ S x | t j > } .Recall (Fact 3.8) that a point x ∈ ∆ belongs to ∂ Ω Vin if and only if W S x is infinite.Consider the following two conditions (negating conditions ¬ (IC) of Section 1.3and ¬ (ZT) of Theorem 1.8): (IC) there exist disjoint subsets S (cid:48) , S (cid:48)(cid:48) of S such that W S (cid:48) and W S (cid:48)(cid:48) are both infiniteand commute; (ZT) there exists an irreducible standard subgroup W S (cid:48) of W S with (cid:59) (cid:54)= S (cid:48) ⊂ S suchthat the Cartan submatrix A W S (cid:48) is of zero type.The goal of this section is to establish the following. Proposition 5.6.
In the setting of Theorem 5.2, we have Σ ∩ ∂ Ω Vin (cid:54)= (cid:59) if and only if (IC) or (ZT) holds. More precisely,(1) suppose that there exists [ x ] ∈ Σ ∩ ∂ Ω Vin and write x = (cid:80) Nj = t j v j with t ∈ (cid:82) N (cid:202) ; theneither W S + , t and W S tx are both infinite and commute (hence (IC) holds), or S + , tx has an irreducible component S (cid:48) such that the Cartan submatrix A W S (cid:48) is of zerotype (hence (ZT) holds);(2) conversely, suppose that (IC) or (ZT) holds with W S (cid:48) irreducible, and let t = ( t j ) s j ∈ S (cid:48) be a Perron–Frobenius vector of − A W S (cid:48) as in Section 3.5; then x : = (cid:80) s j ∈ S (cid:48) t j v j ∈ V satisfies [ x ] ∈ Σ ∩ ∂ Ω Vin . Recall from Section 3.1 the notion of irreducible components of a Coxeter group. Asin Section 1.3, for any T ⊂ S we set T ⊥ : = { s i ∈ S (cid:224) T | m i , j = ∀ s j ∈ T } . For two subsets T , T ⊂ S , we write T ⊥ T when T ⊂ T ⊥ (i.e. m i , j = s i ∈ T and s j ∈ T ). Ourproof of Proposition 5.6 relies on the following lemma. Lemma 5.7.
In the setting of Theorem 5.2, if x = (cid:80) Nj = t j v j satisfies t ∈ (cid:82) N (cid:202) and [ x ] ∈ Σ ,then(1) S + , t ⊥ S tx ;(2) S + , t has no irreducible component S (cid:48) such that the Cartan submatrix A W S (cid:48) is ofpositive type;(3) S + , tx has no irreducible component S (cid:48) such that the Cartan submatrix A W S (cid:48) is ofnegative type.In particular, W S + , t is infinite and W S x = W S tx × W S + , tx .Proof of Lemma 5.7. By definition, t j (cid:202) α i ( x ) (cid:201) (cid:201) i , j (cid:201) N .(1) Let us check that any element of S tx commutes with any element of S + , t . Forany s i ∈ S x we have by definition(5.6) 0 = α i ( x ) = n (cid:88) j = t j α i ( v j ) = (cid:88) s j ∈ S + , t t j A i , j .If s i ∈ S tx , then s i ∉ S + , t , hence A i , j (cid:201) s j ∈ S + , t ; thus each term of the right-hand sum in (5.6) is nonpositive, hence must be zero. Thus for any s i ∈ S tx and s j ∈ S + , t we have A i , j =
0, which means that s i and s j commute.(2) Suppose by contradiction that S has an irreducible component S (cid:48) such that theCartan submatrix A W S + , t is of positive type. By Fact 3.15, we may assume that A W S (cid:48) is symmetric, and positive definite. The vector x (cid:48) : = ( t i ) s i ∈ S (cid:48) has all its entries positive,and since t i (cid:202) α i ( x ) (cid:201) s i ∈ S we have0 (cid:202) (cid:88) s i ∈ S (cid:48) t i α i ( x ) = x (cid:48) t A W S (cid:48) x (cid:48) ,which contradicts the fact that the symmetric matrix A W S (cid:48) is positive definite.(3) Let S (cid:48) be an irreducible component of S . For any s i ∈ S (cid:48) ,0 = α i ( x ) = (cid:88) s j ∈ S (cid:48) t j A i , j + (cid:88) s j ∈ S (cid:224) S (cid:48) t j A i , j ;since t j A i , j (cid:201) s j ∈ S (cid:224) S (cid:48) , we have (cid:80) s j ∈ S (cid:48) t j A i , j (cid:202)
0. Thus the vector x (cid:48) = ( t j ) s j ∈ S (cid:48) has only positive entries and A W S (cid:48) x (cid:48) = ( (cid:80) s j ∈ S (cid:48) t j A i , j ) s i ∈ S (cid:48) has only nonnegative entries.By Fact 3.14, the Cartan submatrix A W S (cid:48) is not of negative type. (cid:3) Proof of Proposition 5.6. (1) Since [ x ] ∈ ∂ Ω Vin , the group W S x is infinite by Fact 3.8,hence W S tx is infinite or W S + , tx is infinite by Lemma 5.7. If W S tx is infinite, then W S + , t and W S tx are both infinite and commute by Lemma 5.7.(1)–(2). Otherwise W S + , tx isinfinite, and then S + , tx has an irreducible component S (cid:48) such that the Cartan submatrix A W S (cid:48) is of zero type, since negative type is excluded by Lemma 5.7.(3). ONVEX COCOMPACTNESS FOR COXETER GROUPS 33 (2) Suppose (IC) holds: there exist disjoint subsets S (cid:48) , S (cid:48)(cid:48) ⊂ S such that W S (cid:48) and W S (cid:48)(cid:48) are both infinite and commute. Up to replacing S (cid:48) by a smaller subset, we mayassume that W S (cid:48) is irreducible. Since W S (cid:48) is infinite, the Cartan submatrix A W S (cid:48) isnot of positive type. As in Section 3.5, let t = ( t j ) s j ∈ S (cid:48)(cid:48) be a Perron–Frobenius vector of2 Id − A W S (cid:48)(cid:48) . Then t is a positive vector and A W S (cid:48) t (cid:201)
0. Let x : = (cid:80) s j ∈ S (cid:48) t j v j . For s i ∈ S (cid:48)(cid:48) we have α i ( x ) =
0, for s i (cid:54)∈ S (cid:48) ∪ S (cid:48)(cid:48) we have α i ( x ) (cid:201)
0, and for s i ∈ S (cid:48) we have α i ( x ) = (cid:88) s j ∈ S (cid:48) t j α i ( v j ) = A W S (cid:48) t (cid:201) S (cid:48)(cid:48) ⊂ S x and [ x ] ∈ Σ . Since W S (cid:48)(cid:48) is infinite, Fact 3.8 implies [ x ] ∈ ∂ Ω Vin .Suppose (ZT) holds: there exists an irreducible standard subgroup W S (cid:48) with S (cid:48) ⊂ S such that A W S (cid:48) is of zero type. Let t = ( t j ) s j ∈ S (cid:48) be a Perron–Frobenius vector of A (cid:48) W S (cid:48) = − A W S (cid:48) . Then t is a positive vector and A W S (cid:48) t =
0. Let x : = (cid:80) s j ∈ S (cid:48) t j v j . For s i (cid:54)∈ S (cid:48) wehave α i ( x ) (cid:201)
0, and for s i ∈ S (cid:48) we have α i ( x ) = (cid:88) s j ∈ S (cid:48) t j α i ( v j ) = A W S (cid:48) t = S (cid:48) ⊂ S x and [ x ] ∈ Σ . Since W S (cid:48) is infinite, Fact 3.8 implies [ x ] ∈ ∂ Ω Vin . (cid:3)
6. P
ROOF OF T HEOREM (CC) ⇒ (NCC) of Theorem 1.8 is immediate from the definitions.The equivalence ¬ (ZT) ⇔ ¬ (ZD) is contained in Fact 3.15. We now establish the otherimplications of Theorem 1.8.6.1. The affine case.
When W S is affine, Theorem 1.8 is contained in the following. Proposition 6.1.
Let W S be an affine irreducible Coxeter group and ρ : W S → GL( V ) arepresentation of W S as a reflection group with Cartan matrix A . Then the followingare equivalent:(1) ρ ( W S ) is convex cocompact in (cid:80) ( V ) ;(2) ρ ( W S ) is naively convex cocompact in (cid:80) ( V ) ;(3) ρ ( W S ) does not contain any unipotent element;(4) the Cartan matrix A is not of zero type;(5) det A (cid:54)= ;(6) W S is of type (cid:101) A N − where N (cid:202) (see Table 2) and the Cartan matrix A is ofnegative type. In this case, if ρ ( W S ) is reduced and dual-reduced, then N = dim( V ) and Ω Vin is asimplex in (cid:80) ( V ), divided by ρ ( W S ): see Example 3.10 if N = N (cid:202) Proof.
The implication (1) ⇒ (2) is immediate from the definitions. For (2) ⇒ (3), seeProposition 2.8. For (4) ⇒ (5) ⇒ (6), see Fact 3.15. For (3) ⇒ (4), note that ρ α ( W S ) ⊂ GL( V α ) is reduced, where ρ α : W S → GL( V α ) isthe representation of W S as a reflection group in V α induced by ρ (see Remarks 3.6and 3.16). If A is of zero type, then by Fact 3.15, the group ρ α ( W S ) acts properlydiscontinuously and cocompactly on some affine chart of (cid:80) ( V α ), preserving some Eu-clidean metric; in particular, ρ α ( W S ) contains a translation of this affine chart, i.e. aunipotent element of GL( V α ). Since the action of ρ ( W S ) on V α is trivial, ρ ( W S ) containsa unipotent element of GL( V ).Let us check (6) ⇒ (1). Suppose W S is of type (cid:101) A N − and A is of negative type. ByFact 3.15, we have det A (cid:54)=
0, hence V v ∩ V α = { } . Let ρ v : W S → GL( V v ) = GL( V α v ) be therepresentation of W S as a reflection group induced by ρ , as in Remarks 3.6 and 3.16; itis reduced and dual-reduced. Let Ω v Vin ⊂ (cid:80) ( V v ) be the corresponding Vinberg domain.By Lemma 3.17, the group ρ v ( W S ) divides Ω v Vin , hence it is convex cocompact in (cid:80) ( V v ).Therefore ρ ( W S ) is convex cocompact in (cid:80) ( V ) by Lemma 6.2. (cid:3) Preliminary reductions.
For any Coxeter group W S and any representation ρ : W S → GL( V ) of W S as a reflection group, associated to some α ∈ V ∗ N and v ∈ V N , wedenote by ρ α : W S → GL( V α ) and ρ α v : W S → GL( V α v ) the induced representations of W S as in Remark 3.6; they have the same Cartan matrix A as ρ . If A is of negative type,then ρ α and ρ α v are still representations of W S as a reflection group by Remark 3.16,hence injective. Lemma 6.2.
Let W S be an irreducible Coxeter group and ρ : W S → GL( V ) a representa-tion of W S as a reflection group, associated to some α ∈ V ∗ N and v ∈ V N , with a Cartanmatrix A of negative type. Then(1) ρ ( W S ) is convex cocompact (resp. strongly convex cocompact) in (cid:80) ( V ) if and onlyif ρ α v ( W S ) is convex cocompact (resp. strongly convex cocompact) in (cid:80) ( V α v ) ;(2) if ρ ( W S ) is naively convex cocompact in (cid:80) ( V ) , then ρ α v ( W S ) is naively convex co-compact in (cid:80) ( V α v ) .Proof. (1) We refer to the matrices (3.10): by Proposition 2.10, the group ρ ( W S ) is con-vex cocompact in (cid:80) ( V ) if and only if ρ α ( W S ) is convex cocompact in (cid:80) ( V α ) which, byCorollary 2.11, happens if and only if ρ α v ( W S ) is convex cocompact in (cid:80) ( V α v ). The simi-lar statement on strong convex cocompactness follows by Proposition 2.7.(2) Let Ω be a ρ ( W S )-invariant properly convex open subset of (cid:80) ( V ) and C ⊂ Ω anonempty ρ ( W S )-invariant closed convex subset which has compact quotient by ρ ( W S ).If N =
2, then W S is affine and we conclude using Proposition 6.1. We now assume N (cid:202)
3. Then Ω ⊂ Ω Vin by Proposition 4.1.(3).We first note that ρ v ( W S ) is naively convex cocompact in (cid:80) ( V v ). Indeed, the set C ∩ (cid:80) ( V v ) is nonempty by Corollary 3.19.(2). It is a ρ v ( W S )-invariant closed subsetof the ρ v ( W S )-invariant properly convex open subset Ω ∩ (cid:80) ( V v ) of (cid:80) ( V v ), and the actionof ρ v ( W S ) on it is cocompact since it is a closed subset of C and C is cocompact.We next check that ρ α ( W S ) is naively convex cocompact in (cid:80) ( V α ). Every pointof (cid:80) ( V α ) has infinite stabilizer (all of W S ), hence (cid:80) ( V α ) ∩ Ω Vin = (cid:59) by Fact 3.8. Let
ONVEX COCOMPACTNESS FOR COXETER GROUPS 35 π : (cid:80) ( V ) (cid:224) (cid:80) ( V α ) → (cid:80) ( V α ) be the natural projection. The projection π ( C ) is a ρ α ( W S )-invariant closed convex subset of the ρ α ( W S )-invariant properly convex open subset π ( Ω ) of (cid:80) ( V α ). The image ∆ α ⊂ (cid:80) ( V α ) of the fundamental polyhedral cone (cid:101) ∆ α for ρ α ( W S )satisfies π − ( ∆ α ) = ∆ (cid:224) (cid:80) ( V α ). In particular, π ( C ) ∩ ∆ α = π ( C ∩ ∆ ) is a compact funda-mental domain for the action of ρ α ( W S ) on π ( C ).Since ρ v ( W S ) is naively convex cocompact in (cid:80) ( V v ) and since ( V v ) α = V v ∩ V α implies( V v ) α = V α v , we get that the group ρ α v ( W S ) is naively convex cocompact in (cid:80) ( V α v ) byapplying the previous reasoning to ρ v ( W S ). (cid:3) A convex subset C of Ω Vin with compact quotient by ρ ( W S ) , for large W S . We now establish the equivalence (NCC) ⇔ ¬ (ZT) of Theorem 1.8 in the case thatthe irreducible Coxeter group W S is large and that V α = { } and V v = V , i.e. ρ ( W S ) isreduced and dual-reduced.In this case, by Proposition 4.1, the Vinberg domain Ω Vin is properly convex andcontains all other ρ ( W S )-invariant open convex subsets of (cid:80) ( V ). Let Ω min be the small-est nonempty ρ ( W S )-invariant convex open subset of Ω Vin , as given by Proposition 2.6.By Theorem 5.2 and Lemma 5.3, the set Ω min = ρ ( W S ) · Σ (cid:91) is the interior of ρ ( W S ) · Σ ,where Σ (cid:91) ⊂ Σ ⊂ (cid:80) ( V ) are the projections of the sets (cid:101) Σ (cid:91) ⊂ (cid:101) Σ ⊂ V of (5.2)–(5.3), obtainedby pruning (cid:101) ∆ . Consider the following nonempty ρ ( W S )-invariant closed convex subsetof Ω Vin :(6.1) C : = Ω min ∩ Ω Vin . Proposition 6.3.
Let W S be a large irreducible Coxeter group and ρ : W S → GL( V ) arepresentation of W S as a reflection group such that ρ ( W S ) is reduced and dual-reduced.Then the following are equivalent:(1) Σ ⊂ Ω Vin ;(2) the set C of (6.1) has compact quotient by ρ ( W S ) ;(3) ρ ( W S ) is naively convex compact in (cid:80) ( V ) ;(4) the following conditions both hold: ¬ (IC) there do not exist disjoint subsets S (cid:48) , S (cid:48)(cid:48) of S such that W S (cid:48) and W S (cid:48)(cid:48) are bothinfinite and commute; ¬ (ZT) for any irreducible standard subgroup W S (cid:48) of W S with (cid:59) (cid:54)= S (cid:48) ⊂ S, the Cartansubmatrix A W S (cid:48) : = ( A i , j ) s i , s j ∈ S (cid:48) is not of zero type. The implication (4) ⇒ (1) of Proposition 6.3 was established by the first three authors[DGK1, Lem. 8.9] when W S is right-angled and ρ preserves a symmetric bilinear form,and by the last two authors [LM, Lem. 4.8] in general. Proof.
We first check the implication (1) ⇒ (2). Using the notation of Section 5.1, theinterior of D is contained in O Vin = ( Ω min ) ∗ , hence Ω min ⊂ D ∗ , hence C ∩ ∆ ⊂ D ∗ ∩ ∆ = Σ .Therefore, taking ρ ( W S )-orbits, the set C is contained in C Σ : = (cid:83) γ ρ ( γ ) · Σ . The set Σ iscompact. If Σ ⊂ Ω Vin , then the action of ρ ( W S ) on C Σ ⊂ Ω Vin is properly discontinuousand cocompact, and the same holds for C since C is a closed subset of C Σ . We now check (2) ⇒ (1). By Theorem 5.2, the convex set Σ (cid:91) is a fundamental domainfor the action of ρ ( W S ) on Ω min , and so Σ (cid:91) = ∆ ∩ Ω min . If C = Ω min ∩ Ω Vin has compactquotient by ρ ( W S ), then the closed set ∆ ∩ C is contained in Ω Vin . But Σ = Σ (cid:91) , hence Σ ⊂ Ω Vin .The implication (2) ⇒ (3) holds by Definition 1.1 of naive convex cocompactness.The implication (3) ⇒ (2) holds because any nonempty ρ ( W S )-invariant closed convexsubset C (cid:48) has to contain Ω min , hence C by Proposition 2.6. The equivalence (1) ⇔ (4)is contained in Proposition 5.6. (cid:3) Cocompact convex sets are large enough.
We now establish the implication (NCC) ⇒ (CC) of Theorem 1.8 in the case that the irreducible Coxeter group W S islarge and ρ ( W S ) is reduced and dual-reduced. We use the following terminology. Definition 6.4 ([DGK2, Def. 1.19]) . Let C be a properly convex subset of (cid:80) ( V ) withnonempty interior. The ideal boundary of C is ∂ i C : = C (cid:224) C . The nonideal boundary of C is ∂ n C : = C (cid:224) C ◦ .The implication (NCC) ⇒ (CC) is contained in the following. Proposition 6.5.
Let W S be a large irreducible Coxeter group and ρ : W S → GL( V ) a representation of W S as a reflection group such that ρ ( W S ) is reduced and dual-reduced. Suppose there is a ρ ( W S ) -invariant properly convex open subset Ω of (cid:80) ( V ) anda nonempty ρ ( W S ) -invariant closed convex subset C of Ω which has compact quotientby ρ ( W S ) . Then Λ orb Ω ( ρ ( W S )) ⊂ ∂ i C , hence ρ ( W S ) is convex cocompact in (cid:80) ( V ) .Proof. By Proposition 4.1.(3), the set Ω is contained in the Vinberg domain Ω Vin , and sowe may assume Ω = Ω Vin . Suppose by contradiction that Λ orb Ω Vin ( ρ ( W S )) (cid:54)⊂ ∂ i C . Let C bethe convex hull of ∂ i C in C and, for t >
0, let C t be the closed uniform t -neighborhoodof C in Ω Vin with respect to the Hilbert metric d Ω Vin . The set C t is properly convex[Bu, (18.12)]; it is ρ ( W S )-invariant and has compact quotient by ρ ( W S ).Since Λ orb Ω Vin ( ρ ( W S )) (cid:54)⊂ ∂ i C , there exists y ∈ Ω Vin whose ρ ( W S )-orbit admits an accu-mulation point ζ ∈ ∂ Ω Vin (cid:224) ∂ i C ; necessarily y ∉ C . Let s = d Ω Vin ( y , C ) >
0. Let E s bethe convex hull of ∂ i C s in Ω Vin ; it is a closed ρ ( W S )-invariant subset of C s , hence ithas compact quotient by ρ ( W S ). The set ∂ i E s = ∂ i C s contains ζ , hence is strictly largerthan ∂ i C . Therefore E s is strictly larger than C and ∂ n E s contains a point not in C .Since E s has compact quotient by ρ ( W S ), there is a point y ∈ ∂ n E s achieving maximumdistance 0 < t (cid:201) s to C . By maximality of t , we have E s ⊂ C t . Let H y be a hyperplanesupporting C t (and therefore also E s ) at y . The intersection C (cid:48) : = H y ∩ E s is a nonemptyconvex set which is the convex hull of some subset of ∂ i E s . Observe that C (cid:48) is containedin ∂ n C t and is therefore disjoint from C .We claim that any hyperplane H supporting C t along C (cid:48) is invariant under thegroup Γ (cid:48) generated by all the reflections ρ ( r ), r ∈ W S , whose fixed hyperplane H ( r )separates C (cid:48) into two connected components. Indeed, consider such a reflection r .First, we note that H ∩ H ( r ) = ( ρ ( r ) · H ) ∩ H ( r ). Second, we show that ρ ( r ) · H also ONVEX COCOMPACTNESS FOR COXETER GROUPS 37 H ( r ) H zx (cid:48) x C (cid:48) Ω Vin E s F IGURE
7. Illustration for the proof of Proposition 6.5contains C (cid:48) . Consider any point x ∈ C (cid:48) (cid:224) H ( r ) and choose a second point x (cid:48) ∈ C (cid:48) (cid:224) H ( r )on the opposite side of H ( r ) (see Figure 7). The segment [ x , x (cid:48) ] crosses H ( r ) at somepoint z ∈ ( x , x (cid:48) ). Since ρ ( r ) fixes z , and H is a supporting hyperplane to C t ⊃ ρ ( r ) · [ x , x (cid:48) ]at z , it follows that ρ ( r ) · [ x , x (cid:48) ] ⊂ H , in particular ρ ( r ) · x ∈ H , hence x ∈ ρ ( r ) · H . Since H is spanned by H ∩ H ( r ) and C (cid:48) , we deduce that ρ ( r ) · H = H . Therefore H is invariantunder the group Γ (cid:48) generated by the set of such reflections, as claimed.In particular, the convex sets C (cid:48) = H y ∩ E s and Ω (cid:48) : = H y ∩ Ω Vin of H y are invariantunder Γ (cid:48) . Moreover, since the action of ρ ( W S ) on E s is cocompact, the intersection of C (cid:48) with the tiling of Ω Vin decomposes C (cid:48) into compact polytopes, hence the action of Γ (cid:48) on C (cid:48) is also cocompact. In particular, since C (cid:48) is noncompact, Γ (cid:48) must be infinite.The above construction finds a proper subspace (cid:80) ( V (cid:48) ) : = (cid:80) (span( C (cid:48) )), an infinite sub-group Γ (cid:48) of ρ ( W S ) generated by reflections and preserving (cid:80) ( V (cid:48) ), and a closed convexsubset C (cid:48) ⊂ Ω (cid:48) : = (cid:80) ( V (cid:48) ) ∩ Ω Vin such that • C (cid:48) is the convex hull of a closed Γ (cid:48) -invariant subset of ∂ Ω (cid:48) , and • the action of Γ (cid:48) on C (cid:48) is cocompact, but • C (cid:48) is disjoint from C .To find a contradiction, consider (cid:80) ( V (cid:48) ), Γ (cid:48) < ρ ( W S ), and C (cid:48) ⊂ Ω (cid:48) = (cid:80) ( V (cid:48) ) ∩ Ω Vin satis-fying the above and so that the dimension of (cid:80) ( V (cid:48) ) is minimal. There are two cases toconsider: ( i ) C (cid:48) = Ω (cid:48) and ( ii ) C (cid:48) (cid:54)= Ω (cid:48) .In case ( i ), the group Γ (cid:48) acts on Ω (cid:48) cocompactly. If Λ Γ (cid:48) denotes the proximal limitset of Γ (cid:48) in (cid:80) ( V (cid:48) ), then the convex hull Conv( Λ Γ (cid:48) ) of Λ Γ (cid:48) in Ω (cid:48) is equal to Ω (cid:48) = C (cid:48) byFact 2.4 and Corollary 3.19.(1). Therefore C (cid:48) = Conv( Λ Γ (cid:48) ) ⊂ Conv( Λ ρ ( W S ) ) ⊂ C , whereConv( Λ ρ ( W S ) ) is the convex hull of Λ ρ ( W S ) in Ω Vin , contradicting the fact that C (cid:48) isdisjoint from C .In case ( ii ), there exists a point x ∈ ∂ n C (cid:48) ⊂ Ω (cid:48) and a supporting hyperplane H x of C (cid:48) at x (in (cid:80) ( V (cid:48) )). Then (cid:80) ( V (cid:48)(cid:48) ) : = (cid:80) ( V (cid:48) ) ∩ H x , C (cid:48)(cid:48) : = H x ∩ C (cid:48) , and Ω (cid:48)(cid:48) : = H x ∩ Ω (cid:48) are invariant under the group Γ (cid:48)(cid:48) generated by all the reflections ρ ( r ) whose fixed hyperplane H ( r )separates C (cid:48)(cid:48) into two connected components and the Γ (cid:48)(cid:48) -action on C (cid:48)(cid:48) is cocompact.However, (cid:80) ( V (cid:48)(cid:48) ) = (cid:80) ( V (cid:48) ) ∩ H x has dimension one less than that of (cid:80) ( V (cid:48) ), contradictingminimality. (cid:3) Proof of Theorem 1.8.
When the infinite irreducible Coxeter group W S is affine,Theorem 1.8 is contained in Proposition 6.1. We now assume that W S is large, satisfy-ing ¬ (IC) and ( (cid:101) A) . Let ρ α v : W S → GL( V α v ) be the representation of W S as a reflectiongroup induced by ρ as in Remarks 3.6 and 3.16; the group ρ α v ( W S ) is reduced anddual-reduced. The equivalences (NCC) ⇔ (CC) ⇔ ¬ (ZT) ⇔ ¬ (ZD) then follow fromLemma 6.2, Propositions 6.3 and 6.5, and Fact 3.15, as in the following diagram: ρ ( W S ) (NCC) in (cid:80) ( V ) ρ α v ( W S ) (NCC) in (cid:80) ( V α v ) ¬ (ZT) ρ ( W S ) (CC) in (cid:80) ( V ) ρ α v ( W S ) (CC) in (cid:80) ( V α v ) ¬ (ZD) . by definition Lemma 6.2.(2) Proposition 6.5 Proposition 6.3Fact 3.15Lemma 6.2.(1)
7. P
ROOF OF T HEOREM
AND CONSEQUENCES
Proof of Theorem 1.3.
Suppose there exist a finite-dimensional real vector space V and a representation ρ ∈ Hom ref ( W S , GL( V )) such that ρ ( W S ) is naively convex co-compact in (cid:80) ( V ). By Lemma 6.2.(2), we may assume that ρ ( W S ) is reduced and dual-reduced. By Propositions 6.1 and 6.3, condition ¬ (IC) holds. Moreover, condition ( (cid:101) A) also holds because for any affine irreducible Coxeter group which is not of type (cid:101) A k , thecorresponding Cartan matrix is of zero type by Fact 3.15.Conversely, suppose conditions ¬ (IC) and ( (cid:101) A) hold. Let N = S be the number ofgenerators of W S . Consider the matrix ( − π / m i , j )) (cid:201) i , j (cid:201) N of W S , and modify it intoa matrix A ∈ M N ( (cid:82) ) in the following way: • for each pair ( i , j ) with m i , j = ∞ , replace the entry − π / m i , j ) = − < − • let P be the set of pairs ( i , j ) with i < j and m i , j =
3. For each ( i , j ) ∈ P , choose anumber t i , j >
1, in such a way that for any disjoint subsets P (cid:48) and P (cid:48)(cid:48) of P , wehave (cid:89) ( i , j ) ∈ P (cid:48) t i , j × (cid:89) ( i , j ) ∈ P (cid:48)(cid:48) t − i , j (cid:54)= i , j ) ∈ P , multiply the ( i , j )-entry − π / m i , j ) = − t i , j and the ( j , i )-entry − π / m j , i ) = − t − i , j to obtain the matrix A .By construction of A , all Cartan submatrices of A corresponding to subgroups W S (cid:48) of type (cid:101) A k are nonsymmetrizable, hence A does not have any Cartan submatrix ofzero type. By [Vi1, Cor. 1], there exists a representation ρ ∈ Hom ref ( W S , GL( (cid:82) N )) withCartan matrix A . By Theorem 1.8, the group ρ ( W S ) is naively convex cocompact in (cid:80) ( (cid:82) N ). This completes the proof of Theorem 1.3. ONVEX COCOMPACTNESS FOR COXETER GROUPS 39
Consequences of Theorems 1.3 and 1.8.
We now prove Corollaries 1.7, 1.10,1.11, and 1.13.
Proof of Corollary 1.7.
Assume there exist V and ρ ∈ Hom ref ( W S , GL( V )) such that ρ ( W S )is naively convex cocompact in (cid:80) ( V ). By Theorem 1.3, the Coxeter group W S must sat-isfy condition ¬ (IC) . The relative hyperbolicity of W S then follows from a theorem ofCaprace [Ca1, Cor. D]: the Coxeter group W S is relatively hyperbolic with respect to acollection P of virtually abelian subgroups of rank at least 2 if and only if for any dis-joint subsets S (cid:48) , S (cid:48)(cid:48) of S with W S (cid:48) and W S (cid:48)(cid:48) both infinite and commuting, the subgroup W S (cid:48) ∪ S (cid:48)(cid:48) is virtually abelian. This criterion is vacuously satisfied, due to ¬ (IC) .It only remains to explain why the subgroups in P are of the form claimed. By [Ca1,Th. B], every P ∈ P is conjugate to one of finitely many standard subgroups { W T i } (cid:201) i (cid:201) l ,each of which is virtually abelian and maximal for this property; and each virtuallyabelian subgroup of W S is conjugate to a subgroup of some W T i .Fix 1 (cid:201) i (cid:201) l . The Coxeter group W T i is a product of irreducible standard Cox-eter groups. Since W T i is virtually abelian, each irreducible factor is either affineor spherical (see Section 3.4). Since affine irreducible Coxeter groups are infinite,condition ¬ (IC) implies that W T i has exactly one affine irreducible factor W U . Fur-ther W U is of type (cid:101) A k for k (cid:202) W S satisfies condition ( (cid:101) A) of Theorem 1.3.Hence, W T i = W U × W U ⊥ where W U ⊥ is the standard subgroup of W S generated by U ⊥ : = { s ∈ S | m u , s = ∀ u ∈ U } . Necessarily, W U ⊥ is a product of irreducible sphericalCoxeter subgroups. (cid:3) Proof of Corollary 1.10.
Suppose ρ ( W S ) is convex cocompact in (cid:80) ( V ) and let W S (cid:48) be aninfinite standard subgroup of W S . By Theorem 1.8, for any (cid:59) (cid:54)= S (cid:48)(cid:48) ⊂ S , the Cartansubmatrix A W S (cid:48)(cid:48) : = ( A i , j ) s i , s j ∈ S (cid:48)(cid:48) is not of zero type. In particular, this holds for any (cid:59) (cid:54)= S (cid:48)(cid:48) ⊂ S (cid:48) . Therefore, if W S (cid:48) is an irreducible Coxeter group, then Theorem 1.8 yieldsthat ρ ( W S (cid:48) ) is convex cocompact in (cid:80) ( V ).In general, W S (cid:48) may not be irreducible, but since W S satisfies condition ¬ (IC) ofTheorem 1.3 we know that W S (cid:48) has a finite-index subgroup W S (cid:48)(cid:48) which is standardand irreducible. Then ρ ( W S (cid:48)(cid:48) ) is convex cocompact in (cid:80) ( V ), and so ρ ( W S (cid:48) ) is convexcocompact in (cid:80) ( V ) (see [DGK2, Lem. 11.3]). (cid:3) Proof of Corollary 1.11. If ρ ( W S ) is strongly convex cocompact in (cid:80) ( V ), then W S is wordhyperbolic by Theorem 2.7, and A i , j A j , i > i (cid:54)= j with m i , j = ∞ by Theorem 1.8.Conversely, suppose W S is word hyperbolic and A i , j A j , i > i (cid:54)= j with m i , j = ∞ .Since W S is word hyperbolic, it does not contain any subgroup isomorphic to (cid:90) , andso conditions ¬ (IC) and ( (cid:101) A) of Theorem 1.3 hold. Condition ¬ (ZD) of Theorem 1.8is also satisfied because for any i (cid:54)= j with m i , j = ∞ , the Cartan submatrix (cid:161) A i , j A j , i (cid:162) is not of zero type if and only if A i , j A j , i >
4. Thus ρ ( W S ) is convex cocompact in (cid:80) ( V ) byTheorem 1.8, and strongly convex cocompact in (cid:80) ( V ) by Theorem 2.7. (cid:3) Proof of Corollary 1.13.
The implications (SCC) ⇒ (CC) ⇒ (NCC) hold by definition,and the equivalence (SCC) ⇔ (WH+) holds by Corollary 1.11. We now prove theimplication (NCC) ⇒ (SCC) .We first note that, by Moussong’s hyperbolicity criterion [Mo] (see Remark 1.6), aCoxeter group with no affine irreducible standard subgroup in three or more generatorsis word hyperbolic if and only if there do not exist disjoint subsets S (cid:48) and S (cid:48)(cid:48) of S suchthat W S (cid:48) and W S (cid:48)(cid:48) are both infinite and commute.Suppose that (NCC) holds. Item ¬ (ZD) of Theorem 1.8, and Fact 3.15, show that W S does not contain any affine irreducible standard subgroup in three or more generators,except possibly of type (cid:101) A k . The latter are ruled out by assumption of Corollary 1.13,and so W S does not contain any affine irreducible standard subgroup in (cid:202) W S is word hyperbolic since W S must satisfy condition ¬ (IC) of Theorem 1.3. Finally, ρ ( W S ) is convex cocompact in (cid:80) ( V ) by Theorem 1.8,and strongly convex cocompact in (cid:80) ( V ) by Theorem 2.7. (cid:3) Remark 7.1.
In [DGK1, § 8], the first three authors proved Corollary 1.13 in the spe-cial case that n = N and that the Cartan matrix A = ( A i , j ) (cid:201) i , j (cid:201) N for ρ is symmetricand defines a nondegenerate quadratic form on V as in Remark 3.11. This approachwas used by the last two authors [LM] to construct the first examples of discrete sub-groups of O( p , 2) which are strongly convex cocompact in (cid:80) ( (cid:82) p + ) and whose (proximal)limit set is homeomorphic to the ( p − p , 1) as abstract groups.8. T HE DEFORMATION SPACE OF CONVEX COCOMPACT REPRESENTATIONS
We now prove Corollaries 1.15, 1.16, and 1.18.8.1.
Convex cocompactness and the interior of χ ref ( W S , GL( V )) . Proof of Corollary 1.15.
By Remark 3.4, the space of characters of W S defined by data( α , v ) satisfying (3.2) is an open subset of χ ( W S , GL( V )) containing χ ref ( W S , GL( V )). ByFact 3.22, the map assigning the conjugacy class of Cartan matrix to a conjugacy classof semisimple representations is a homeomorphism from the space consisting of repre-sentations defined by data ( α , v ) satisfying (3.2) to the space of N × N matrices of rankat most dim( V ) that are weakly compatible with W S (Definition 3.1), considered up toconjugation by positive diagonal matrices.The reflection characters χ ref ( W S , GL( V )) correspond to the subset defined by (3.6),namely that A i , j A i , j (cid:202) m i , j = ∞ , and (3.5), namely that the cone (cid:101) ∆ = { x ∈ V | α i ( x ) (cid:201) ∀ (cid:201) i (cid:201) N } has nonempty interior. Note that the validity of (3.5)is almost automatic since W S is word hyperbolic. Indeed, if N (cid:202) W S is large,hence the Cartan matrix A W S is of negative type by Fact 3.15, and so (3.5) holds asexplained in Remark 3.13.(1). If N =
2, then (3.5) may fail only when A W S is of zerotype, which is equivalent to A A = ONVEX COCOMPACTNESS FOR COXETER GROUPS 41
In any case, the reflection characters which are convex cocompact in (cid:80) ( V ) are an opensubset of χ ( W S , GL( V )) which, by Theorem 1.8, corresponds to the subset of Cartanmatrices compatible with W S defined by the strict inequalities A i , j A j , i > m i , j = ∞ . Let us now check that they are precisely the interior of χ ref ( W S , GL( V )) ifdim( V ) (cid:202) N .Consider a semisimple representation ρ : W S → GL( V ) of W S as a reflection groupwhich is not convex cocompact in (cid:80) ( V ). Then the associated Cartan matrix A satisfies A i , j A j , i = i , j with m i , j = ∞ . Deforming the entry A i , j to become less neg-ative gives a matrix A remaining within the space of weakly compatible matrices, forwhich A i , j A j , i becomes smaller than 4. However, under the assumption dim( V ) (cid:202) N ,such a small deformation of the Cartan matrix corresponds to a small deformation in χ ( W S , GL( V )) which is outside of χ ref ( W S , GL( V )). This shows that the character of ρ isnot in the interior of χ ref ( W S , GL( V )). (cid:3) Convex cocompactness and the interior of
Hom ref ( W S , GL( V )) . As discussedin the introduction, it is a much more subtle problem to determine the interior of thespace Hom ref ( W S , GL( V )) which includes many nonsemisimple representations. Proof of Corollary 1.16.
Since convex cocompactness in (cid:80) ( V ) is an open condition [DGK2,Th. 1.17], the set of representations ρ ∈ Hom ref ( W S , GL( V )) for which ρ ( W S ) is convexcocompact in (cid:80) ( V ) is included in the interior of Hom ref ( W S , GL( V )) in Hom( W S , GL( V )).Let us prove the reverse inclusion.Suppose ρ ∈ Hom ref ( W S , GL( V )) has Cartan matrix A satisfying A i , j A j , i = i , j ) such that m i , j = ∞ , and let us find conditions for the existence of a smalldeformation of ρ outside Hom ref ( W S , GL( V )). The pair ( i , j ) being fixed, define(8.1) V i : = span m i , k <∞ ( v k ) ⊂ V , W (cid:63) i : = Ann( V i ) = (cid:84) m i , k <∞ Ann( v k ) ⊂ V ∗ , V (cid:63) j : = span m k , j <∞ ( α k ) ⊂ V ∗ , W j : = Ann( V (cid:63) j ) = (cid:84) m k , j <∞ Ker( α k ) ⊂ V .If there exists ( β , w ) ∈ W (cid:63) i × W j such that β ( w ) =
1, then for any t ∈ (cid:82) the elements α ti : = α i + t β and v tj : = v j + tw satisfy α ti ( v k ) = α i ( v k ) for all k with m i , k (cid:54)= ∞ , and α k ( v tj ) = α k ( v j ) for all k with m k , j (cid:54)= ∞ ; but α ti ( v tj ) = α i ( v j ) + t ( λ + t ) where λ = α i ( w ) + β ( v j ). If t isnonzero, of the sign of λ when λ (cid:54)=
0, then α ti ( v tj ) > α i ( v j ). Replacing ( α i , v j ) with ( α ti , v tj )thus yields a representation ρ t whose Cartan matrix A t satisfies A ti , j = α ti ( v tj ) > A i , j and A tj , i = A j , i , hence A ti , j A tj , i < A i , j A j , i =
4. Thus ρ t ∉ Hom ref ( W S , GL( V )) for such t ,but ρ t → t → ρ .In the two cases, we will show that there must exist a pair ( β , w ) ∈ W (cid:63) i × W j such that β ( w ) =
1. Indeed, suppose not. This means that:(8.2) W (cid:63) i ⊂ Ann( W j ) = V (cid:63) j . First, we assume that n (cid:202) N −
2. We have dim( V (cid:63) j ) (cid:201) N − W (cid:63) i ) (cid:202) n − ( N − (cid:202) N −
1. Note also that α j ∈ V (cid:63) j and α j ∉ W (cid:63) i because A j , i (cid:54)=
0. So, W (cid:63) i is a strict subspaceof V (cid:63) j , which implies dim W (cid:63) i (cid:201) N −
2. Contradiction.Finally, we assume that W is right-angled and n (cid:202) N . Let R (cid:96) = { k | m (cid:96) , k = } and R i , j = R i ∩ R j . Let r (cid:96) = R (cid:96) and r i , j = R i , j . Then(8.3) (cid:189) dim V (cid:63) j (cid:201) + r j ,dim W (cid:63) i (cid:202) n − − r i (since dim V i (cid:201) + r i ).By the pigeonhole principle, since the sets R i and R j are disjoint from { i , j } in {
1, . . . , N } ,we have(8.4) r i , j (cid:202) r i + r j − ( N − W is word hyperbolic, for any k , (cid:96) ∈ R i , j the generators s k , s (cid:96) must commute(otherwise s k s (cid:96) and s i s j generate a copy of (cid:90) ). Hence, the pairing V (cid:63) j × V i → (cid:82) has rank > r i , j , because its restriction to indices ( R i , j (cid:116) { j } ) × ( R i , j (cid:116) { i } ) has matrixDiag(2, . . . , 2, A j , i ). But the pairing W (cid:63) i × V i → (cid:82) is zero by (8.1), and so the inclusion(8.2) forces dim V (cid:63) j − dim W (cid:63) i > r i , j . Using (8.3) and (8.4), we obtain n < N . (cid:3) Remark 8.1.
Consider the map A : V N × ( V ∗ ) N → M N ( (cid:82) ) sending (( v k ) Nk = , ( α k ) Nk = ) tothe matrix ( α i ( v j )) Ni , j = . In the case that v , . . . , v N are linearly independent in V (resp. α , . . . , α N are linearly independent in V ∗ ), the entries of the matrix A (( v k ) Nk = , ( α k ) Nk = )may each be perturbed independently by perturbing the linear forms ( α k ) Nk = (resp. byperturbing the vectors ( v k ) Nk = ); thus A is an open map near (( v k ) Nk = , ( α k ) Nk = ) and theconclusion of Corollary 1.16 is immediate. However, A fails to be an open map nearsome inputs (( v k ) Nk = , ( α k ) Nk = ) for which both ( v k ) Nk = and ( α k ) Nk = are linearly dependent.In some degenerate cases outside of the context of reflection groups, this may happeneven for dim V as large as 2 N −
2. This explains why the proof above of Corollary 1.16is needed.
Remark 8.2.
In the case that n : = dim V < N , it is a priori possible that the conclusionof Corollary 1.16 could fail. Specifically, we cannot rule out that some product A i , j A j , i is constant equal to 4 on an irreducible component of Hom ref ( W S , GL( V )) nor that A i , j A j , i achieves the value 4 as a local minimum in the interior of Hom ref ( W S , GL( V )).We are not aware of an example demonstrating such behavior where W S is word hy-perbolic and right-angled, but there exist examples where W S is right-angled (see Ex-ample 8.4) or word hyperbolic (see Example 8.5).8.3. The Anosov condition and the interior of
Hom ref ( W S , GL( V )) . Here is a con-sequence of Fact 1.17.
Lemma 8.3.
Let W S be an infinite, word hyperbolic, irreducible Coxeter group and let ρ : W S → GL( V ) be a representation of W S as a reflection group. Then ρ is P -Anosov ifand only if ρ ( W S ) is strongly convex cocompact in (cid:80) ( V ) . Corollary 1.18 follows directly from Corollaries 1.11 and 1.16, Proposition 2.7, andLemma 8.3.
ONVEX COCOMPACTNESS FOR COXETER GROUPS 43
Proof of Lemma 8.3. If ρ ( W S ) is strongly convex cocompact in (cid:80) ( V ), then ρ is P -Anosovby Fact 1.17. Conversely, suppose ρ is P -Anosov. We cannot immediately applyFact 1.17 because ρ ( W S ) might not preserve a properly convex open subset of (cid:80) ( V ).However, consider the induced representation ρ α v : W S → GL( V α v ) as in Remarks 3.6and 3.16, with the same Cartan matrix A as ρ . It is easy to check (see [GW, Prop. 4.1]and (3.10)) that ρ α v : W S → GL( V α v ) is still P -Anosov. The representation ρ α v is reduced(Definition 3.5), hence preserves a properly convex open subset of (cid:80) ( V α v ), namely theVinberg domain for ρ α v ( W S ) (Proposition 4.1.(1)). By Fact 1.17, the group ρ α v ( W S ) isconvex cocompact in (cid:80) ( V α v ). Note that the Cartan matrix A is of negative type: thisfollows from Proposition 6.1 if N =
2, and from the fact that the infinite, word hyper-bolic, irreducible Coxeter group W S is large if N = ρ ( W S ) isconvex cocompact in (cid:80) ( V ) by Lemma 6.2.(1). (cid:3) Three examples.
First, we give an example of a right-angled but not word hyper-bolic Coxeter group in 5 generators such that the cyclic products A A and A A take only the value 4 on all of Hom ref ( W S , GL( (cid:82) )). Example 8.4.
Let W S be the right-angled Coxeter group in N = s s s s s ∞ ∞ ∞ ∞ Since there exist disjoint subsets S (cid:48) = { s , s } and S (cid:48)(cid:48) = { s , s } of S such that W S (cid:48) and W S (cid:48)(cid:48) are both infinite and commute, the group W S contains a subgroup isomorphic to (cid:90) ,hence is not word hyperbolic. Any compatible Cartan matrix for W S must be conjugate,by diagonal matrices, to A = − x − − y − − z
00 0 − − u − for some x , y , z , u (cid:202)
1. The (2, 4) − minor of A is 16, hence A always has rank (cid:202) x = y = z = u =
1. Since the rank is 4, this Cartan matrix is alsorealized as the Cartan matrix for a reflection group in V for dim V =
4: for example,let V be the span of the columns of A inside (cid:82) . A further linear algebra calculationshows that if rank A =
4, then x = u =
1. Indeed, it is an exercise to show that det( A ) =
32 ( xu + xz + yu − x − y − z − u + = x = u =
1, using the inequalities x , y , z , u (cid:202)
1. Hence if dim V =
4, then all products A A and A A take only thevalue 4 on all of Hom ref ( W S , GL( V )), which, up to conjugation, is two-dimensional.Second, we give an example of a word hyperbolic but not right-angled Coxeter groupin 6 generators such that the set of characters [ ρ ] ∈ χ ref ( W S , GL( (cid:82) )) for which ρ ( W S )is convex cocompact in (cid:80) ( (cid:82) ) is not the interior of χ ref ( W S , GL( (cid:82) )) in χ ( W S , GL( (cid:82) )). In particular, the set of representations ρ ∈ Hom ref ( W S , GL( (cid:82) )) for which ρ ( W S ) is convexcocompact in (cid:80) ( (cid:82) ) is not the interior of Hom ref ( W S , GL( (cid:82) )) in Hom( W S , GL( (cid:82) )). Example 8.5.
Let W S be the Coxeter group in N = s s s s s s
66 66 ∞ We can easily check by Moussong’s hyperbolicity criterion (see Remark 1.6) that theCoxeter group W S is word hyperbolic, and see that any compatible Cartan matrix for W S must be conjugate, by diagonal matrices, to A = −(cid:112) −(cid:112) x −(cid:112) −(cid:112) −(cid:112) −(cid:112) −(cid:112) x − −(cid:112) − y
00 0 0 − y −(cid:112)
30 0 0 0 −(cid:112) for some x > y (cid:202)
1. The (1, 1) − minor of A is − y + (cid:54)=
0, hence A always hasrank (cid:202)
5. In particular, all representations of W S as a reflection group in (cid:82) must be ir-reducible, hence discussing character or equality up to conjugation is equivalent. Rankequal to 5 is achieved by the Cartan matrix A with det( A ) = y − x + x − ) − = V =
5, then the space of representations in Hom ref ( W S , GL( V )) up to conju-gation is homeomorphic to the line (cid:82) . Moreover, the intersection of Hom ref ( W S , GL( V ))in those coordinates with the line { y = } is reduced to the point ( x , y ) = (1, 1). Hence,by Theorem 1.8, the line Hom ref ( W S , GL( V )) except one point corresponds to convexcocompact representations (see Figure 8).Lastly, we give an example of a not word hyperbolic and not right-angled Cox-eter group in 4 generators such that the set of characters [ ρ ] ∈ χ ref ( W S , GL( (cid:82) )) forwhich ρ ( W S ) is convex cocompact in (cid:80) ( (cid:82) ) is not the interior of χ ref ( W S , GL( (cid:82) )) in χ ( W S , GL( (cid:82) )). Example 8.6.
Let W S be the Coxeter group in N = s s s s Since there is a subset S (cid:48) = { s , s , s } of S with W S (cid:48) of type (cid:101) A , the group W S containsa subgroup isomorphic to (cid:90) , hence is not word hyperbolic. We can easily see that any ONVEX COCOMPACTNESS FOR COXETER GROUPS 45 xy (1, 1) F IGURE
8. The curve det( A ) = x > y (cid:202) W S must be conjugate, by diagonal matrices, to A = − − −(cid:112) − − x − − x − − y −(cid:112) − y − for some x , y >
0. A simple calculation shows thatdet( A ) = − (cid:179) x + x − ) + (cid:112) y + y − ) + (cid:112) x y + ( x y ) − ) + (cid:180) < W S as a reflection group in (cid:82) must be irreducible,hence discussing character or equality up to conjugation is equivalent. If dim V = ref ( W S , GL( V )) up to conjugation is homeo-morphic to (cid:82) > ; by Theorem 1.8, convex cocompact representations correspond exactlyto the complement of the line { x = } in (cid:82) > . A PPENDIX
A. T
HE SPHERICAL AND AFFINE C OXETER DIAGRAMS
For the reader’s convenience, we reproduce below the list of all spherical and allaffine irreducible diagrams.For spherical Coxeter groups, the index (in particular the n for types A n , B n , D n ) isthe number of nodes of the diagram, and the standard representation acts irreduciblyon the n -dimensional sphere with a spherical simplex as a fundamental domain.For affine Coxeter groups, the index (in particular the n for types (cid:101) A n , (cid:101) B n , (cid:101) C n , (cid:101) D n )is one less than the number of nodes; the group acts cocompactly on Euclidean n -spacewith a Euclidean simplex as a fundamental domain. A n ( n (cid:202) B n ( n (cid:202) D n ( n (cid:202) I ( p ) ( p (cid:202) p H H F E E E T ABLE
1. The diagrams ofthe irreducible sphericalCoxeter groups (cid:101) A n ( n (cid:202) (cid:101) B n ( n (cid:202) (cid:101) C n ( n (cid:202) (cid:101) D n ( n (cid:202) (cid:101) A ∞ (cid:101) B = (cid:101) C (cid:101) G (cid:101) F (cid:101) E (cid:101) E (cid:101) E T ABLE
2. The diagrams ofthe affine irreducible Cox-eter groups
ONVEX COCOMPACTNESS FOR COXETER GROUPS 47 R EFERENCES [BDL] S. A. B
ALLAS , J. D
ANCIGER , G.-S. L EE , Convex projective structures on non-hyperbolic three-manifolds , Geom. Topol. 22 (2018), p. 1593–1646. 2, 3[Ba] T. B
ARBOT , Deformations of Fuchsian AdS representations are quasi-Fuchsian , J. DifferentialGeom. 101 (2015), p. 1–46. 2[BM] T. B
ARBOT , Q. M
ÉRIGOT , Anosov AdS representations are quasi-Fuchsian , Groups Geom.Dyn. 6 (2012), p. 441–483. 2[Be1] Y. B
ENOIST , Propriétés asymptotiques des groupes linéaires , Geom. Funct. Anal. 7 (1997), p. 1–47. 12[Be2] Y. B
ENOIST , Automorphismes des cônes convexes , Invent. Math. 141 (2000), p. 149–193. 12, 13[Be3] Y. B
ENOIST , Convexes hyperboliques et quasiisométries , Geom. Dedicata 122 (2006) p. 109–134. 2[Be4] Y. B
ENOIST , Convexes divisibles IV , Invent. Math. 164 (2006), p. 249–278. 2, 3[Be5] Y. B
ENOIST , A survey on divisible convex sets , Adv. Lect. Math. 6 (2008), p. 1–18. 2, 3[BPS] J. B
OCHI , R. P
OTRIE , A. S
AMBARINO , Anosov representations and dominated splittings , J.Eur. Math. Soc. 21 (2019), p. 3343–3414. 8[Bo] N. B
OURBAKI , Groupes et algèbres de Lie, Chapitres IV, V, VI , Hermann, 1968. 14[Bu] H. B
USEMANN , The geometry of geodesics , Academic Press Inc., New York, 1955. 36[Ca1] P.-E. C
APRACE , Buildings with isolated subspaces and relatively hyperbolic Coxeter groups ,Innov. Incidence Geom. 10 (2009), p. 15–31. 5, 39[Ca2] P.-E. C
APRACE , Erratum to “Buildings with isolated subspaces and relatively hyperbolic Cox-eter groups” , Innov. Incidence Geom. 14 (2015), p. 77–79. 5[CL] S. C
HOI , G.-S. L EE , Projective deformations of weakly orderable hyperbolic Coxeter orbifolds ,Geom. Topol. 19 (2015), p. 1777–1828. 2[CLM1] S. C
HOI , G.-S. L EE , L. M ARQUIS , Convex projective generalized Dehn filling , Ann. Sci. Éc.Norm. Sup. 53 (2020), p. 217–266. 2, 3[CLM2] S. C
HOI , G.-S. L EE , L. M ARQUIS , Deformation spaces of Coxeter truncation polytopes , inpreparation. 2[CM] M. C
RAMPON , L. M
ARQUIS , Finitude géométrique en géométrie de Hilbert , with an appendixby the authors and C. Vernicos, Ann. Inst. Fourier 64 (2014), p. 2299–2377. 2[DGK1] J. D
ANCIGER , F. G
UÉRITAUD , F. K
ASSEL , Convex cocompactness in pseudo-Riemannian hy-perbolic spaces , Geom. Dedicata 192 (2018), p. 87–126, special issue
Geometries: A Celebrationof Bill Goldman’s 60th Birthday . 2, 5, 7, 8, 9, 10, 11, 35, 40[DGK2] J. D
ANCIGER , F. G
UÉRITAUD , F. K
ASSEL , Convex cocompact actions in real projective geome-try , preprint, arXiv:1704.08711. 1, 2, 3, 7, 8, 9, 10, 11, 12, 13, 26, 36, 39, 41[DGK3] J. D
ANCIGER , F. G
UÉRITAUD , F. K
ASSEL , Examples and counterexamples of convex cocompactgroups , in preparation. 2, 3[Da] M. W. D
AVIS , The geometry and topology of Coxeter groups , Princeton University Press, Prince-ton, NJ, 2008. 31[DH] M. D
ESGROSEILLIERS , F. H
AGLUND , On some convex cocompact groups in real hyperbolicspace , Geom. Topol. 17 (2013), p. 2431–2484. 2, 6[FK] T. F
OERTSCH , A. K
ARLSSON , Hilbert metrics and Minkowski norms , J. Geom. 83 (2005), p. 22–31. 11[GGKW] F. G
UÉRITAUD , O. G
UICHARD , F. K
ASSEL , A. W
IENHARD , Anosov representations and properactions , Geom. Topol. 21 (2017), p. 485–584. 8[GW] O. G
UICHARD , A. W
IENHARD , Anosov representations : Domains of discontinuity and appli-cations , Invent. Math. 190 (2012), p. 357–438. 8, 43[Gu] Y. G
UIVARC ’ H , Produits de matrices aléatoires et applications aux propriétés géométriques dessous-groupes du groupe linéaire , Ergodic Theory Dynam. Systems 10 (1990), p. 483–512. 12[Kac] V. K AC , Infinite-dimensional Lie algebras , Cambridge University Press, Cambridge, 1990. 30[KV] V. K AC , E. B. V INBERG , Quasihomogeneous cones , Math. Notes 1 (1967) p. 231–235. 2[KLP1] M. K
APOVICH , B. L
EEB , J. P
ORTI , Anosov subgroups: Dynamical and geometric characteriza-tions , Eur. J. Math. 3 (2017), p. 808–898. 8 [KLP2] M. K
APOVICH , B. L
EEB , J. P
ORTI , Some recent results on Anosov representations , Transform.Groups 21 (2016), p. 1105–1121. 8[Kas] F. K
ASSEL , Geometric structures and representations of discrete groups , in
Proceedings of theInternational Congress of Mathematicians 2018 (ICM 2018) , p. 1113–1150, World Scientific,2019. 9[KL] B. K
LEINER , B. L
EEB , Rigidity of invariant convex sets in symmetric spaces , Invent. Math. 163(2006), p. 657–676. 2[Kr] D. K
RAMMER , The conjugacy problem for Coxeter groups , PhD thesis, Universiteit Utrecht,1994, published in Groups Geom. Dyn. 3 (2009), p. 71–171. 5[La] F. L
ABOURIE , Anosov flows, surface groups and curves in projective space , Invent. Math. 165(2006), p. 51–114. 8[LM] G.-S. L EE , L. M ARQUIS , Anti-de Sitter strictly GHC-regular groups which are not lattices ,Trans. Amer. Math. Soc. 372 (2019), p. 153–186. 2, 5, 7, 9, 35, 40[MV] G. A. M
ARGULIS , E. B. V
INBERG , Some linear groups virtually having a free quotient , J. LieTheory 10 (2000), p. 171–180. 17, 19[Ma1] L. M
ARQUIS . Espace des modules de certains polyèdres projectifs miroirs , Geom. Dedicata 147(2010), p. 47–86. 2, 3[Ma2] L. M
ARQUIS , Coxeter group in Hilbert geometry , Groups Geom. Dyn. 11 (2017), p. 819–877. 2,6, 19, 20[Me] G. M
ESS , Lorentz spacetimes of constant curvature (1990), Geom. Dedicata 126 (2007), p. 3–45.2, 26[Mo] G. M
OUSSONG , Hyperbolic Coxeter groups , PhD thesis, Ohio State University, 1987. 5, 40[P] R. P
OTRIE , Robust dynamics, invariant structures and topological classification , in
Proceed-ings of the International Congress of Mathematicians 2018 (ICM 2018) , p. 2057–2080, WorldScientific, 2019. 9[Q] J.-F. Q
UINT , Groupes convexes cocompacts en rang supérieur , Geom. Dedicata 113 (2005), p. 1–19. 2[Sa] L. S
ANDGREN , On convex cones , Math. Scand. 2 (1954), p. 19–28. 28[Se] E. S
ENETA , Non-negative matrices and Markov chains , revised reprint of the second (1981)edition, Springer Series in Statistics, Springer, New York, 2006. 18[Ve] J. V EY , Sur les automorphismes affines des ouverts convexes saillants , Ann. Sc. Norm. Super.Pisa 24 (1970), p. 641–665. 12[Vi1] E. B. V
INBERG , Discrete linear groups generated by reflections , Math. USSR Izv. 5 (1971),p. 1083–1119. 2, 4, 14, 16, 17, 18, 19, 20, 22, 23, 38[Vi2] E. B. V
INBERG , Absence of crystallographic groups of reflections in Lobachevski˘ı spaces of largedimension , Trudy Moskov. Mat. Obshch. 47 (1984), p. 68–102. 2[Z] A. Z
IMMER , Projective Anosov representations, convex cocompact actions, and rigidity , J. Dif-ferential Geom., to appear; arXiv:1704.08582. 2, 8, 9
ONVEX COCOMPACTNESS FOR COXETER GROUPS 49 D EPARTMENT OF M ATHEMATICS , T HE U NIVERSITY OF T EXAS AT A USTIN , 1 U
NIVERSITY S TATION
C1200, A
USTIN , TX 78712, USA
Email address : [email protected] CNRS
AND L ABORATOIRE P AUL P AINLEVÉ , U
NIVERSITÉ L ILLE
1, 59655 V
ILLENEUVE D ’A SCQ C EDEX , F
RANCE
Email address : [email protected] CNRS
AND L ABORATOIRE A LEXANDER G ROTHENDIECK , I
NSTITUT DES H AUTES É TUDES S CIEN - TIFIQUES , U
NIVERSITÉ P ARIS -S ACLAY , 35
ROUTE DE C HARTRES , 91440 B
URES - SUR -Y VETTE , F
RANCE
Email address : [email protected] D EPARTMENT OF M ATHEMATICS , S
UNGKYUNKWAN U NIVERSITY , 2066 S
EOBU - RO , J ANGAN - GU ,S UWON , 16419, S
OUTH K OREA
Email address : [email protected] U NIV R ENNES , CNRS, IRMAR - UMR 6625, F-35000 R
ENNES , F
RANCE
Email address ::