Orbit Closures and Invariants
aa r X i v : . [ m a t h . AG ] D ec ORBIT CLOSURES AND INVARIANTS
MICHAEL BATE, HARALAMPOS GERANIOS, AND BENJAMIN MARTIN
Abstract.
Let G be a reductive linear algebraic group, H a reductive subgroup of G and X an affine G -variety. Let X H denote the set of fixed points of H in X , and N G ( H ) the normalizer of H in G . Inthis paper we study the natural map of quotient varieties ψ X,H : X H /N G ( H ) → X/G induced by theinclusion X H ⊆ X . We show that, given G and H , ψ X,H is a finite morphism for all affine G -varieties X if and only if H is a G -completely reducible subgroup of G (in the sense defined by J-P. Serre); thiswas proved in characteristic 0 by Luna in the 1970s. We discuss some applications and give a criterionfor ψ X,H to be an isomorphism. We show how to extend some other results in Luna’s paper to positivecharacteristic and also prove the following theorem. Let H and K be reductive subgroups of G ; thenthe double coset HgK is closed for generic g ∈ G if and only if H ∩ gKg − is reductive for generic g ∈ G . Introduction
The purpose of this paper is to establish some results in geometric invariant theory over fields ofpositive characteristic, where tools from characteristic 0—such as Luna’s ´Etale Slice Theorem—are notavailable. In particular, we prove the following theorem and give some applications (see Section 2 forprecise definitions of terms). Let k be an algebraically closed field of characteristic p ≥ Theorem 1.1.
Suppose G is a reductive linear algebraic group over k and H is a reductive subgroup of G . Then the following are equivalent: (i) H is G -completely reducible; (ii) N G ( H ) is reductive and, for every affine G -variety X , the natural map of quotients ψ X,H : X H /N G ( H ) → X/G is a finite morphism (here X H denotes the H -fixed points in X ). The study of closed orbits is central in geometric invariant theory—the closed orbits for G in X parametrise the points of the quotient variety X/G . An important piece of the proof of Theorem 1.1 isProposition 4.1, which gives a connection between the closed G -orbits in X and the closed H -orbits in X ; cf. [33], [48], [2] and [7], for example.Theorem 1.1 reduces to the main result in Luna’s paper [33] when k has characteristic 0, becausecondition (i) and the first hypothesis of (ii) are automatic in characteristic 0 if H is already assumed tobe reductive. Luna’s methods use the powerful machinery of ´etale slices, based on his celebrated “´EtaleSlice Theorem” [32]; see Section 3.1 below for more on ´etale slices. Many useful consequences flow fromthe existence of an ´etale slice (see Proposition 3.1 below, for example). Although ´etale slices sometimesexist in positive characteristic [1], in general they do not. Our methods differ from Luna’s in that theyapply equally well in all characteristics. These methods also allow us to provide extensions to positivecharacteristic of other results from [33] (see Proposition 3.10, Remark 4.2(i) and Proposition 4.7).The work in this paper fits into a broad continuing programme of taking results about algebraicgroups and related structures from characteristic zero and proving analogues in positive characteristic.A basic problem with this process is that results—such as the existence of an ´etale slice—that are truewhen p = 0 may simply fail when p > reductive if it has trivial unipotent radical, and linearly reductive ifall its rational representations are semisimple). When p = 0, a connected group is linearly reductive ifand only if it is reductive, whereas if p > R G is finitely generated, where R is a finitely-generated k -algebra and G ⊆ Aut( R ) is reductive. This was resolved in characteristic 0 Date : December 7, 2018.2010
Mathematics Subject Classification.
Key words and phrases.
Geometric invariant theory; quotient variety; G -complete reducibility; ´etale slice; double cosets. n the 1950s, but not in positive characteristic until the 1970s (see the introduction to Haboush’s paper[22]).In some contexts in positive characteristic where the hypothesis of reductivity is too weak and linearreductivity is too strong, it has been found that a third notion, that of G -complete reducibility, providesa good balance (cf. [37, Cor. 1.5]) and our main theorem is another example of this phenomenon. SeeSection 2.4 for the definition. The idea is that when p = 0 there is no distinction between demanding thata subgroup H of a reductive group G is reductive or linearly reductive or G -completely reducible, butthere is a huge difference in positive characteristic. The notion of complete reducibility was introducedby Serre [52] and Richardson [49], and over the past twenty years or so has found many applications inthe theory of algebraic groups, their subgroup structure and representation theory, geometric invarianttheory, and the theory of buildings: for examples, see [2], [5], [9], [30], [35], [36], [39], [56], [57], [58].The paper is set out as follows. Section 2 contains preparatory material from geometric invarianttheory and the theory of complete reducibility. The proof of Theorem 1.1 contains three main ingredients,each dealt with in a separate section. In Section 3 we build on work of Bardsley and Richardson toestablish the important technical result Proposition 3.10, which gives a criterion for a map of quotientvarieties to be finite. In Section 4 we carry out our analysis of the closed G - and H -orbits and showthat ψ X,H is quasi-finite if H is G -completely reducible (Theorem 4.4). In Section 5 we show that theimage of ψ X,H is closed (Theorem 5.1). The key idea here is to consider the map of projectivisations P ( X H ) → P ( X ) induced by the inclusion of X H in X when X is a G -module; the G -complete reducibilityof H guarantees that we get a well-defined map of quotient varieties P ( X H ) /G → P ( X ) /G . Section 6draws these strands together and completes the proof of Theorem 1.1 using Proposition 3.8 (a variationon Zariski’s Main Theorem).Section 7 gives a criterion for ψ X,H to be an isomorphism onto its image (Theorem 7.2). In Section 8we use representation theory to construct some examples relevant to Theorem 1.1. In Section 9 wegive a criterion (Theorem 9.1) for generic double cosets
HgK of G to be closed, where H and K arereductive subgroups of G . Luna proved a stronger result [31] in characteristic 0 using ´etale slice methods,but our techniques work when ´etale slices are not available. We give some applications of Theorem 9.1(Examples 9.11 and 9.12); these serve as applications of Theorem 1.1 as well. We finish in Section 10 byusing the theory we have developed to prove some results on complete reducibility. Acknowledgements : The first author would like to thank Sebastian Herpel for the conversations we hadwhich led to the first iteration of some of the ideas in this paper, and also Stephen Donkin for some veryhelpful nudges towards the right literature. All three authors acknowledge the funding of EPSRC grantEP/L005328/1. We would like to thank the anonymous referee for their very insightful comments andfor pointing out a subtle gap in the proof of Theorem 1.1.2.
Notation and Preliminaries
Notation.
Our basic references for the theory of linear algebraic groups are the books [10] and[54]. Unless otherwise stated, we work over a fixed algebraically closed field k with no restriction on thecharacteristic. By a variety we mean a quasi-projective variety over k , and we identify a variety X withits set of k -points. For a linear algebraic group G over k , we let G denote the connected component of G containing the identity element 1 and R u ( G ) E G denote the unipotent radical of G . We say that G is reductive if R u ( G ) = { } ; note that we do not require a reductive group to be connected. Whenwe discuss subgroups of G , we really mean closed subgroups; for two such subgroups H and K of G , weset HK := { hk | h ∈ H, k ∈ K } . We denote the centralizer of a subgroup H of G by C G ( H ), and thenormalizer by N G ( H ). All group actions are left actions unless otherwise indicated.We make repeated use of the following result [35, Lem. 6.8]: if G is reductive and if H is a reductivesubgroup of G then N G ( H ) = H C G ( H ) .Given a linear algebraic group G , let Y ( G ) denote the set of cocharacters of G , where a cocharacter is a homomorphism of algebraic groups λ : k ∗ → G . Note that since the image of a cocharacter isconnected, we have Y ( G ) = Y ( G ). A linear algebraic group G acts on its set of cocharacters: for g ∈ G , λ ∈ Y ( G ) and a ∈ k ∗ , we set ( g · λ )( a ) = gλ ( a ) g − .Given an affine variety X over k , we denote the coordinate ring of X by k [ X ] and the function fieldof X (when X is irreducible) by k ( X ). Given x ∈ X , we let T x ( X ) denote the tangent space to X at x . Recall that for a linear algebraic group G , T ( G ) has the structure of a Lie algebra, which we also Richardson originally defined strong reductivity for subgroups of G , but his notion was shown to be equivalent toSerre’s in [5, Thm. 3.1]. enote by Lie( G ) or g . Given a morphism φ : X → Y of affine varieties X and Y and a point x ∈ X , welet d x φ : T x ( X ) → T φ ( x ) ( Y ) denote the differential of φ at x . We say that X is a G -variety if the linearalgebraic group G acts morphically on X . If X is affine then the action of G on X gives a linear actionof G on k [ X ], defined by ( g · f )( x ) = f ( g − · x ) for all g ∈ G , f ∈ k [ X ] and x ∈ X . Given a G -variety X and x ∈ X , we denote the G -orbit through x by G · x and the stabilizer of x in G by G x . If x, y ∈ X aretwo points on the same G -orbit, then we sometimes say x and y are G -conjugate . For x ∈ X , we denotethe orbit map G → G · x , g g · x by κ x ; we say the orbit G · x is separable if κ x is separable. We denoteby X G the set of G -fixed points in X , and by k [ X ] G the ring of G -invariant functions in k [ X ].Given a morphism of varieties f : V → W , define e ( v ) for v ∈ V to be max(dim( Z )), where Z rangesover the irreducible components of f − ( f ( v )) that contain v . By [10, AG.10.3], e ( v ) is an upper semi-continuous function of v . This implies the following useful result about dimensions of stabilizers for a G -variety X [44, Lem. 3.7(c)]: for any r ∈ N ∪ { } , the set { x ∈ X | dim( G x ) ≥ r } is closed. We deducethe lower semi-continuity of orbit dimension: that is, for any r ∈ N ∪{ } , the set { x ∈ X | dim( G · x ) ≤ r } is closed. In particular, the set { x ∈ X | dim( G · x ) is maximal } is open. We also need an infinitesimalversion of these results. Given a variety Z , we denote the (reduced) tangent bundle of Z by T Z ; we mayidentify
T Z with the set of pairs { ( z, v ) | z ∈ Z, v ∈ T z ( Z ) } , and we have a canonical embedding from Z to T Z given by z ( z, k ; here we take the tangent bundle to be the corresponding reduced scheme.) If ψ : Z → W is a morphism of varieties then we have a map dψ : T Z → T W given by dψ ( z, v ) = ( ψ ( z ) , d z ψ ( v )). Lemma 2.1.
For any r ∈ N ∪ { } , the set { x ∈ X | dim( G x ) + dim(ker( d κ x )) ≥ r } is closed.Proof. Define α : G × X → X × X by α ( g, x ) = ( g · x, x ). We obtain a morphism dα from T ( G × X ) ∼ = T G × T X to T ( X × X ) ∼ = T X × T X . Let x ∈ X and consider the point y := (( x, , ( x, ∈ T X × T X .Now ( dα ) − ( y ) is a closed subset C y of T G × T X ; it is clear that C y = { (( g, v ) , ( x, | g ∈ G x , v ∈ ker( d κ x )) } . Each irreducible component of this set has dimension e ′ ( y ) := dim( G x ) + dim(ker( d κ x )).Define a function s : X → T G × T X by s ( x ) = ((1 , , ( x, X with a closed subsetof T X × T X via the embedding x (( x, , ( x, s is a morphism, we deduce from the uppersemi-continuity of the function e ( v )—taking ( V, W, f ) = (
T G × T X, T X × T X, dα )—that the function e ′ ( y ) is also upper semi-continuous. The result now follows. (cid:3) A morphism φ : X → Y of affine varieties is said to be finite if the coordinate ring k [ X ] is integralover the image of the comorphism φ ∗ : k [ Y ] → k [ X ]. Finite morphisms are closed [40, Prop. I.7.3(i)]; inparticular, a dominant finite morphism is surjective. A morphism of affine varieties is called quasi-finite if its fibres are finite; finite morphisms are always quasi-finite [40, Prop. I.7.3(ii)], but the converse is nottrue. A dominant morphism φ : X → Y of irreducible varieties is called birational if the comorphisminduces an isomorphism of function fields k ( X ) ∼ = k ( Y ). Given an irreducible affine variety X , wecan form the normalization of X by considering the normal affine variety e X whose coordinate ring isthe integral closure of k [ X ] in the function field k ( X ). The normalization map ν X : e X → X is, byconstruction, finite, birational and surjective. Remark . We record an observation which we use several times in the sequel. Let φ : X → Y and ψ : Y → Z be morphisms of affine varieties with ψ ◦ φ finite. Then it is easy to see that:(i) φ is finite;(ii) if φ is dominant then ψ is finite.We say that a property P ( x ) holds for generic x ∈ X if there is an open dense subset U of X suchthat P ( x ) holds for all x ∈ U .For the remainder of the paper, we fix the convention that G denotes a reductive linear algebraicgroup over k .2.2. Group actions and quotients.
The main result of this paper, Theorem 1.1, concerns quotientsof affine varieties by reductive algebraic group actions. Let X be an affine G -variety. As noted above, G acts on k [ X ], and we can form the subring k [ X ] G ⊆ k [ X ] of G -invariant functions on X . It follows from[43] and [22] that k [ X ] G is finitely generated, and hence we can form an affine variety denoted X/G withcoordinate ring k [ X/G ] = k [ X ] G . Moreover, the inclusion k [ X ] G ֒ → k [ X ] gives rise to a morphism from X to X/G , which we shall denote by π X,G : X → X/G . The map π X,G has the following properties [41,Thm. A.1.1], [44, Thm. 3.5], [1, § π X,G is surjective; ii) π X,G is constant on G -orbits in X ;(iii) π X,G separates disjoint closed G -invariant subsets of X ;(iv) each fibre of π X,G contains a unique closed G -orbit, and π X,G determines a bijective map fromthe set of closed G -orbits in X to X/G ;(v)
X/G is a categorical quotient of X : that is, for every variety V and every morphism ψ : X → V which is constant on G -orbits, there is a unique morphism ψ G : X/G → V such that ψ = ψ G ◦ π X,G .(This means π X,G is a good quotient in the sense of [44, Chapter 3, §
4, p57]. More generally, if X is aquasi-projective G -variety and π is a map from X to another quasi-projective variety Y then we call π a good quotient if it is an affine map and satisfies (i)–(v) above.) We say that π X,G : X → X/G is a geometric quotient if the fibres of π X,G are precisely the G -orbits. This is the case if and only if every G -orbit is closed (for instance, if every G -orbit has the same dimension—e.g., if G is finite).If φ : Y → X is a G -equivariant morphism of affine G -varieties, then the restriction of the comorphismto k [ X ] G induces a natural morphism from Y /G to X/G , which we shall denote by φ G . In a special caseof this construction, we have the following result, which follows from [44, Thm. 3.5, Lem. 3.4.1]. Lemma 2.3.
Let X be an affine G -variety and let i : Y → X be an embedding of a closed G -stablesubvariety Y in X . Then π X,G ( Y ) is closed in X/G . Moreover, the induced map i G : Y /G → X/G isinjective and finite.Remark . If char( k ) = 0 then i G is an isomorphism onto its image. This need not be the case inpositive characteristic: see Example 3.2.We record some other useful results. First, note that if G is a finite group, then the map π G aboveis a finite morphism. To see this, let f ∈ k [ X ] and let T be an indeterminate. Then the polynomial F ( T ) := Q g ∈ G ( T − g · f ) ∈ ( k [ X ])[ T ] is monic and has coefficients in k [ X ] G , and F ( f ) = 0. This showsthat k [ X ] is integral over k [ X ] G , which gives the claim.If X is irreducible and normal then X/G is normal [1, 2.19(a)], while if G is connected then k [ X ] G isintegrally closed in k [ X ] [1, 2.4.1].Now suppose H is a subgroup of G such that the normalizer N G ( H ) is reductive. Then the inclusion X H ⊆ X induces a map of quotients ψ X,H : X H /N G ( H ) → X/G . Theorem 1.1 asserts that when H isa G -completely reducible subgroup of G (in the sense of Section 2.4 below), this map is always a finitemorphism.For technical reasons, we sometimes need to work with affine G -varieties satisfying an extra property. Definition 2.5.
Let X be an affine G -variety. We denote by X cl the closure of the set { x ∈ X | G · x is closed } . Following Luna [32, Sec. 4] , we say that X has good dimension (“bonne dimension”)if X cl = X . We say that x is a stable point of X for the G -action if dim( G · x ) is maximal and G · x isclosed [44, Ch. 3, § , [41, Ch. 1, § .Remark . The set of stable points is open [44, Ch. 3, § §
4] (this is true even withoutthe assumption that G is reductive). Since the set { x ∈ X | dim( G · x ) is maximal } is open, it followsthat if X is irreducible then X has good dimension if and only if there exists a stable point. Moreover,if X has good dimension then generic fibres of π X,G : X → G are orbits of G . Hence if X is irreduciblethen dim( X/G ) = dim( X ) − m , where m is the maximal orbit dimension. Lemma 2.7.
Let X be an irreducible affine G -variety with good dimension. Then k ( X/G ) = k ( X ) G .Moreover, π X,G is separable.Proof.
It is clear that k ( X/G ) is a subfield of k ( X ) G . Conversely, let f ∈ k ( X ) G . Set U = { x ∈ X | there exists h , h ∈ k [ X ] such that f = h /h and h ( x ) = 0 } . Then U is a nonempty open subset of X , and clearly U is G -stable. Hence C := X \ U is closed and G -stable. As X has good dimension, there exists 0 = h ∈ k [ X ] G such that h | C = 0. Now f is a globallydefined regular function on the corresponding principal open set X h , so f ∈ k [ X h ] = k [ X ][1 /h ]. Hence f = f ′ h r for some f ′ ∈ k [ X ] and some r ≥
0. Then f ′ is G -invariant, since f is, so f ∈ k ( X/G ).The second assertion is [1, 2.1.9(b)]. Note that separability can fail if X does not have good dimension:see [38]. (cid:3) Lemma 2.8.
Let φ : X → Y be a finite surjective G -equivariant map of affine G -varieties. i) For all x ∈ X , G · x is closed if and only if G · φ ( x ) is closed. Moreover, if y ∈ Y and G · y is closed then φ − ( G · y ) is a finite union of G -orbits, each of which is closed and has the samedimension as G · y . (ii) The map φ G : X/G → Y /G is quasi-finite. (iii) X has good dimension if and only if Y does.Proof. If x ∈ X and G · x is closed then G · φ ( x ) = φ ( G · x ) is closed, as φ is finite. Conversely, let y ∈ Y such that G · y is closed, and let n = dim( G · y ). Let x ∈ φ − ( G · y ). Then dim( G x ) ≤ dim( G y ), sodim( G · x ) ≥ dim( G · y ) = n . But φ is finite, so every irreducible component of φ − ( G · y ) has dimension n . It follows that dim( G · x ) = n and G · x is a union of irreducible components of φ − ( G · y ); in particular, G · x is closed. This proves (i). Part (iii) now follows.To prove part (ii), let x ∈ X , y ∈ Y such that φ G ( π X,G ( x )) = π Y,G ( y ). Without loss of generality, wecan assume that G · x and G · y are closed. Now G · φ ( x ) is closed by (i), so we must have G · φ ( x ) = G · y ,so x ∈ φ − ( G · y ). But φ − ( G · y ) is a finite union of G -orbits by (i), so we are done. (cid:3) Lemma 2.9.
Let φ : X → Y be a finite birational G -equivariant morphism of irreducible affine G -varieties. If one of X or Y has good dimension then φ G : X/G → Y /G is birational.Proof.
By Lemma 2.8(iii), if one of X or Y has good dimension then they both do. It follows fromLemma 2.7 that k ( Y /G ) = k ( X/G ) = k ( X ) G ; hence φ G is birational. (cid:3) Later we also need some material on constructing quotients of projective varieties by actions of re-ductive groups, but we delay this until Section 5.Suppose H is a subgroup of G . Recall that the quotient G/H (which as a set is just the coset space)has the structure of a quasi-projective homogeneous G -variety, and H is the stabilizer of the image of1 ∈ G under the natural map π G,H : G → G/H . Richardson has proved the following in this situation([47, Thm. A]; see also [23]).
Theorem 2.10.
Suppose H is a subgroup of G . Then G/H is an affine variety if and only if H isreductive. Recall also that the Zariski topology on
G/H is the quotient topology: that is, a subset S ⊆ G/H isclosed in
G/H if and only if π − G,H ( S ) is closed in G . We need a technical result. Lemma 2.11.
Let H be a reductive subgroup of G . There exist a G -module Y and a nonempty opensubset U of Y H such that the following hold: (i) G y = H for all y ∈ U ; (ii) G · y is closed for all y ∈ U ; (iii) N G ( H ) · y is closed for all y ∈ U .Proof. Since H is reductive, G/H is affine. The group G acts on G/H by left multiplication. Let x = π G,H (1); then G x = H . If K is a reductive subgroup of G containing H then K · π G,H (1) = π G,H ( K )is closed, as K is a closed subset of G that is stable under right multiplication by H . We can embed G/H equivariantly in a G -module X . By the lower semi-continuity of orbit dimension, there is a nonemptyopen subset U of X H such that dim( G x ) = dim( H ) for all x ∈ U —so G x is a finite extension of H for all x ∈ U . If char( k ) = 0 then we can conclude from Proposition 3.1 that there is an open neighbourhood O of x such that G x ≤ H for all x ∈ O . It then follows (applying the arguments for (ii) and (iii) below)that we can take Y to be X and U a suitable nonempty open subset of X H ∩ O . In general, however,we need a slightly more complicated construction.Let Y be the G -module X ⊕ X . Note that Y H = X H ⊕ X H and for any ( y , y ) ∈ Y H , G ( y ,y ) = G y ∩ G y . We show that Y has the desired properties. For each r ≥
0, define C r = { y ∈ U × U | | G y : H | ≥ r } . Then C r is empty for all but finitely many r by [37, Lem. 2.2 and Defn. 2.3]. Moreover, C r is constructible.For let e C r = { ( y, g , . . . , g r ) | y ∈ U × U , g , . . . , g r ∈ G y , g j g − i H for 1 ≤ i, j ≤ r } ;then C r is the image of e C r under projection onto the first factor. Set D r = C r \ C r +1 . Then the nonempty D r form a finite collection of disjoint constructible sets that cover the irreducible set U × U , so D s contains a nonempty open subset U of U × U for precisely one value of s .We show that s = 1. Suppose not. Choose y = ( x , x ) ∈ U . Let g , g , . . . , g r be coset representativesfor G x /H with g ∈ H . Note that U = { x ∈ U | ( x , x ) ∈ U } is an open dense subset of X H . Let = ( x , x ) ∈ U . Then our hypothesis means that g · ( x , x ) = ( x , x ) for some g H . Now g must fix x , so g ∈ g i H for some i ≥
1; in fact, i ≥ g H . It follows that g i fixes ( x , x ) since H fixes( x , x ), so g i fixes x . But S rj =2 ( X g j ∩ X H ) is a proper closed subset of X H as none of the g j for j ≥ x , so we have a contradiction. We conclude that s = 1 after all. Hence G y = H for all y ∈ U .Set y = ( x , N G ( H ) · x is closed in G/H , so the orbit N G ( H ) · y is closed in Y H .Moreover, N G ( H ) y = H , so N G ( H ) · y has maximal dimension among the N G ( H )-orbits on Y H . Hence y is a stable point of Y H for the N G ( H )-action. A similar argument shows that y is a stable point of G · Y H for the G -action. Since the set of stable points is open in each case, we can find a nonempty opensubset U of U such that (ii) and (iii) hold for U ; then (i) holds for U by construction. This completesthe proof. (cid:3) Cocharacters, G -actions and R-parabolic subgroups. Suppose that X is a G -variety. Forany cocharacter λ ∈ Y ( G ) and x ∈ X we can define a morphism ψ = ψ x,λ : k ∗ → X by ψ ( a ) = λ ( a ) · x for each a ∈ k ∗ . We say that the limit lim a → λ ( a ) · x exists if ψ extends to a morphism ψ : k → X . Ifthe limit exists, then the extension ψ is unique, and we set lim a → λ ( a ) · x = ψ (0). It is clear that, forany G and X , if there exists λ ∈ Y ( G ) such that lim a → λ ( a ) · x exists but lies outside G · x , then G · x is not closed in X .A subgroup P of G is called a parabolic subgroup if the quotient G/P is complete; this is the case ifand only if
G/P is projective. If G is connected and reductive, then all parabolic subgroups of G have a Levi decomposition P = R u ( P ) ⋊ L , where the reductive subgroup L is called a Levi subgroup of P . Inthis case, the unipotent radical R u ( P ) acts simply transitively on the set of Levi subgroups of P , andgiven a maximal torus T of P there exists a unique Levi subgroup of P containing T . For these standardresults see [10], [11] or [54] for example. It is possible to extend these ideas to a non-connected reductivegroup using the formalism of R-parabolic subgroups described in [5, Sec. 6]. We give a brief summary;see loc. cit. for further details. Given a cocharacter λ ∈ Y ( G ), we have:(i) P λ := { g ∈ G | lim a → λ ( a ) gλ ( a ) − exists } is a parabolic subgroup of G ; we call a parabolicsubgroup arising in this way an R-parabolic subgroup of G .(ii) L λ := C G ( λ ) = { g ∈ G | lim a → λ ( a ) gλ ( a ) − = g } is a Levi subgroup of P λ ; we call a Levisubgroup arising in this way an R-Levi subgroup of G .(iii) R u ( P λ ) = { g ∈ G | lim a → λ ( a ) gλ ( a ) − = 1 } .The R-parabolic (resp. R-Levi) subgroups of a connected reductive group G are the same as the parabolicand Levi subgroups of G . Moreover, the results listed above for parabolic and Levi subgroups of connectedreductive algebraic groups also hold for R-parabolic and R-Levi subgroups of non-connected reductivegroups; that is, the unipotent radical R u ( P ) acts simply transitively on the set of R-Levi subgroups ofan R-parabolic subgroup P , and given a maximal torus T of P there exists a unique R-Levi subgroup of P containing T .Now, if H is a reductive subgroup of G and λ ∈ Y ( H ), then λ gives rise in a natural way to R-parabolic and R-Levi subgroups of both G and H . In such a situation, we reserve the notation P λ (resp. L λ ) for R-parabolic (resp. R-Levi) subgroups of G , and use the notation P λ ( H ), L λ ( H ), etc. to denotethe corresponding subgroups of H . Note that for λ ∈ Y ( H ), it is obvious from the definitions that P λ ( H ) = P λ ∩ H , L λ ( H ) = L λ ∩ H and R u ( P λ ( H )) = R u ( P λ ) ∩ H .2.4. G -complete reducibility. Our main result, and many of the intermediate ones, uses the frame-work of G -complete reducibility introduced by J-P. Serre [52], which has been shown to have geometricimplications in [5] and subsequent papers. We give a short recap of some of the key ideas concerningcomplete reducibility.Let H be a subgroup of G . Following Serre (see, for example, [52]), we say that H is G -completelyreducible ( G -cr for short) if whenever H ⊆ P for an R-parabolic subgroup P of G , there exists an R-Levisubgroup L of P such that H ⊆ L . For example, if G = SL n ( k ) or GL n ( k ) then H is G -cr if and only ifthe inclusion of H is completely reducible in the usual sense of representation theory. If H is G -cr then H is reductive, while if H is linearly reductive then H is G -cr (see [5, Sec. 2.4 and Sec. 6]). Hence incharacteristic 0, H is G -cr if and only if H is reductive.In [2] and [37] it was shown that the notion of complete reducibility is useful when one considers G -varieties and, as explained in the introduction, one of the purposes of this paper is to expand uponthis theme.The geometric approach to complete reducibility outlined in [5] rests on the following construction,which was first given in this form in [9]. Given a subgroup H of a reductive group G and a positive nteger n , we call a tuple of elements h ∈ H n a generic tuple for H if there exists a closed embedding of G in some GL m ( k ) such that h generates the associative subalgebra of m × m matrices spanned by H [9, Defn. 5.4]. A generic tuple for H always exists for sufficiently large n . Suppose h ∈ H n is a generictuple for H ; then in [9, Thm. 5.8(iii)] it is shown that H is G -completely reducible if and only if the G -orbit of h in G n is closed, where G acts on G n by simultaneous conjugation.2.5. Optimal cocharacters.
Let X be an affine G -variety. The classic Hilbert-Mumford Theorem [28,Thm. 1.4] says that via the process of taking limits, the cocharacters of G can be used to detect whetheror not the G -orbit of a point in X is closed. Kempf strengthened the Hilbert-Mumford Theorem in [28](see also [24], [41], [51]), by developing a theory of “optimal cocharacters” for non-closed G -orbits. Wegive an amalgam of some results from Kempf’s paper; see [28, Thm. 3.4, Cor. 3.5] (and see also [9, § G ). Theorem 2.12.
Let x ∈ X be such that G · x is not closed, and let S be a closed G -stable subsetof X which meets G · x . Then there exists an R-parabolic subgroup P ( x ) of G and a nonempty subset Ω( x ) ⊆ Y ( G ) such that: (i) for all λ ∈ Ω( x ) , lim a → λ ( a ) · x exists, lies in S , and is not G -conjugate to x ; (ii) for all λ ∈ Ω( x ) , P λ = P ( x ) ; (iii) R u ( P ( x )) acts simply transitively on Ω( x ) ; (iv) G x ⊆ P ( x ) . Preparatory Results
In this section we collect some results concerning algebraic group actions on varieties which will beuseful in the rest of the paper. Recall our standing assumption that G is a reductive group.3.1. ´Etale slices. ´Etale slices are a powerful tool in geometric invariant theory. Let X be an affine G -variety and let x ∈ X such that G · x is closed. Luna introduced the notion of an ´etale slice through x [32, III.1]: this is a locally closed affine subvariety S of X with x ∈ S satisfying certain properties. Heproved that an ´etale slice through x always exists when the ground field has characteristic 0. Bardsleyand Richardson later defined ´etale slices in arbitrary characteristic [1, Defn. 7.1] and gave some sufficientconditions for an ´etale slice to exist [1, Propns. 7.3–7.6]. If an ´etale slice exists through x , the orbit G · x must be separable. We record an important consequence of the ´etale slice theory [1, Prop. 8.6]. Proposition 3.1.
Let X be an affine G -variety and let x ∈ X such that G · x is closed and there isan ´etale slice through x . Then there is an open neighbourhood U of x such that for all u ∈ U , G u isconjugate to a subgroup of G x . The following example, based on a construction from [37, Ex. 8.3], shows that this result need nothold when there is no ´etale slice.
Example . Let G = SL ( k ) and let H = C p × C p = h γ , γ | γ p = γ p = [ γ , γ ] = 1 i . Define f : k × H → k × G by f ( x, h ) = ( x, f x ( h )), where f x ( γ h γ h ) := (cid:18) h x + h x (cid:19) . Set K x = im( f x ).Note that for each x ∈ k , there are only finitely many x ′ ∈ k such that K x and K x ′ are G -conjugate.Define actions of G and H on k × G by g · ( x, g ′ ) = ( x, gg ′ ) and h · ( x, g ′ ) = ( x, g ′ f x ( h ) − ). Theseactions commute with each other, so we have an action of G on the quotient space V := ( k × G ) /H . Set ϕ = π k × G,H . Since H is finite, ϕ is a geometric quotient. A straightforward calculation shows that forany ( x, g ) ∈ k × G , the stabilizer G ϕ ( x,g ) is precisely gK x g − . It follows that the G -orbits on V are allclosed, but the assertion of Proposition 3.1 cannot hold for any v ∈ V . Hence no v ∈ V admits an ´etaleslice. Note that generic stabilizers are nontrivial, but there do exist orbits with trivial stabilizer (take x = 0).Nonetheless we can even show (using ´etale slice methods!) that generic G -orbits in V are separable.Let O = { x ∈ k | x = 0 , x, . . . , ( p − x } , an open subset of k . Then the finite group H acts freely on O × G , so by [1, Prop. 8.2], O × G is a principal H -bundle in the ´etale topology in the sense of [1, Defn.8.1]. Let x ∈ O . It follows that the derivative d ( x,g ) ϕ is surjective for all g ∈ G . Define an H -equivariantmap ψ x : G → k × G by ψ x ( g ) = ( x, g ). An easy computation shows that the map ( ψ x ) H : G/H → V induced by ψ x is bijective and separable when regarded as a map onto its image, so ( ψ x ) H gives byZariski’s Main Theorem an isomorphism from G/H onto its image. Now ( ψ x ) H is G -equivariant, wherewe let G act on G/H by left multiplication. Since π G,H : G → G/H is separable, the orbit G · π G,H ( g ) isseparable for any g ∈ G . This means that the orbit G · ϕ ( x, g ) = G · ( ψ x ) H ( π G,H ( g )) is separable as well. n contrast, consider the orbit G · ϕ (0 , g ). This cannot be separable: for otherwise ϕ (0 , g ) admitsan ´etale slice by [1, Prop. 7.6], since the stabilizer G ϕ (0 ,g ) is trivial, and we know already that this isimpossible. It follows easily that ( ψ ) H : G/H → V is not an isomorphism onto its image. We see fromthis that if i is the obvious inclusion of Y := { } × G in k × G then the induced map i G : Y /H → ( k × G ) /H = V is not an isomorphism onto its image (cf. Remark 2.4).The failure of Proposition 3.1 and other consequences of the machinery of ´etale slices when slices donot exist is behind many of the technical difficulties we need to overcome in order to prove Theorem 1.1.3.2. Some results on closed orbits.
We first need a technical lemma which collects together variousproperties of orbits and quotients and the associated morphisms. For more details, see the proofs of [48,Lem. 4.2, Lem. 10.1.3] or the discussion in [27, Sec. 2.1], for example; the extension to non-connected G is immediate. Note that if G acts on a variety X then for any x ∈ X , G · x is locally closed [10,Prop. 1.8], so it has the structure of a quasi-affine variety. Lemma 3.3.
Let X be a G -variety. Suppose x ∈ X , and let ψ x : G/G x → G · x be the natural map.Then: (i) ψ x is a homeomorphism; (ii) G · x is affine if and only if G/G x is affine if and only if G x is reductive; (iii) ψ x is an isomorphism of varieties if and only if the orbit G · x is separable.Remark . All the subtleties here are only really important in positive characteristic since in character-istic 0 the orbit map is always separable, so the morphism ψ x is always an isomorphism. The result showsthat even in bad cases where the orbit map is not separable we can reasonably compare the quotient G/G x with the orbit G · x , as one might hope. Lemma 3.5.
Let H be a subgroup of G and suppose x ∈ X . Set K = G x and let H act on X byrestriction of the G -action. Then: (i) H · x is closed in G · x if and only if HK = { hk | h ∈ H, k ∈ K } is a closed subset of G . (ii) If G · x is closed in X then H · x is closed in X if and only if HK is closed in G .Proof. Part (ii) follows immediately from part (i). For part (i), since the map ψ x : G/K → G · x fromLemma 3.3 is a homeomorphism, H · x is closed in G · x if and only if the corresponding subset H · π G,K (1)is closed in
G/K (recall that π G,K : G → G/K is the canonical projection). Since
G/K has the quotienttopology, this is the case if and only if the preimage of this orbit is closed in G . But the preimage isprecisely the subset HK . (cid:3) Our next result involves the following set-up: Suppose Y is another G -variety. Then G × G acts onthe product X × Y via ( g , g ) · ( x, y ) = ( g · x, g · y ), and identifying G with its diagonal embedding∆( G ) in G × G , we can also get the diagonal action of G on X × Y : g · ( x, y ) = ( g · x, g · y ). Lemma 3.6.
With the notation just introduced, let x ∈ X , y ∈ Y and set K = G x , H = G y . Then: (i) H · x is closed in G · x if and only if K · y is closed in G · y if and only if G · ( x, y ) is closed in ( G · x ) × ( G · y ) . (ii) If G · x is closed in X and G · y is closed in Y , then H · x is closed in X if and only if K · y isclosed in Y if and only if G · ( x, y ) is closed in X × Y .Proof. (i). The first equivalence follows from Lemma 3.5 since KH = ( HK ) − is closed in G if and onlyif HK is closed in G (note that this argument is based on the one in the proof of [48, Lem. 10.1.4]).For the second equivalence, consider the orbit map κ : G × G → G associated to the orbit of 1 ∈ G forthe double coset action of G × G on G (cf. Section 9); so κ is given by κ ( g , g ) = g g − . Then κ issurjective and open. Now, since the ( G × G )-orbit of ( x, y ) is ( G · x ) × ( G · y ) and the stabilizer of ( x, y )in G × G is K × H , we have that G · ( x, y ) = ∆( G ) · ( x, y ) is closed in ( G · x ) × ( G · y ) if and only if∆( G )( K × H ) is closed in G × G , by Lemma 3.5(i). Now ∆( G )( K × H ) is closed in G × G if and only if( K × H )∆( G ) is, and ( K × H )∆( G ) = κ − ( KH ). Since κ is a surjective open map, we conclude that∆( G )( K × H ) is closed in G × G if and only if KH is closed in G , which happens if and only if K · y isclosed in G · y , by Lemma 3.5(i) again.(ii). This chain of equivalences follows quickly from part (i). (cid:3) Remark . The results above give criteria for a result of the form “ G · x closed implies H · x closed”for a point x in a G -variety X . We can’t hope for a general converse to this. For example, let G be anyconnected reductive group and, in the language of Section 2.4, let x ∈ X = G n be a generic tuple for Borel subgroup of G and y ∈ Y = G n be a generic tuple for G itself. Then, G x = G y = Z ( G ), the G -orbits of y and ( x, y ) are closed, but the G -orbit of x is not closed.3.3. Finite morphisms and quotients.
In this section we provide some general results on finitemorphisms and quotients by reductive group actions. We begin with an extension of Zariski’s MainTheorem which deals with nonseparable morphisms. Recall that if X is an irreducible affine variety then ν X : e X → X denotes the normalization of X . Proposition 3.8.
Let φ : X → Y be a dominant quasi-finite morphism of irreducible affine varieties.Suppose Y is normal and generic fibres of φ are singletons. Then φ is a finite bijection onto an opensubvariety of Y . Moreover, the normalization map ν X : e X → X is a bijection.Proof. As φ is dominant, we may identify k [ Y ] with a subring of k [ X ] and k ( Y ) with a subfield of k ( X ).The hypothesis on the fibres of φ implies that φ is purely inseparable [25, Thm. 4.6]. Let f , . . . , f r be generators for k [ X ] as a k -algebra. Then there exists a power q of p such that f qi ∈ k ( Y ) for all i . Let S be the k -algebra generated by k [ Y ] together with f q , . . . , f qr and let Z be the correspondingaffine variety, so that S = k [ Z ]. Then the inclusions k [ Y ] ⊆ k [ Z ] ⊆ k [ X ] give rise to maps ψ : X → Z and α : Z → Y such that φ = α ◦ ψ . Now k [ X ] is integral over k [ Z ] by construction, so ψ is finite andsurjective, and hence α is quasi-finite and has the same image as φ . But α is birational by construction,so α is an isomorphism from the affine variety Z onto an open subvariety of Y by Zariski’s Main Theorem(since Y is normal). To complete the proof of the first assertion, it is enough to show that ψ is injective.This follows because any k -algebra homomorphism k [ X ] → k is completely determined by its values on f q , . . . , f qr , which are elements of k [ Z ].Because ν X is finite and birational, the map φ ◦ ν X : e X → Y satisfies the hypotheses of the proposition.Hence φ ◦ ν X is injective. This forces ν X to be injective also. But ν X is also surjective, and we aredone. (cid:3) We need some further results about the behaviour of affine G -varieties under normalization. If X isan affine G -variety then e X inherits a unique structure of a G -variety such that ν X is G -equivariant (cf.[1, Sec. 3]). This gives a map of quotients ( ν X ) G : e X/G → X/G . Lemma 3.9.
Let X be an irreducible affine G -variety with good dimension and let ( ν X ) G be as above.Then ( ν X ) G is finite and e X/G is the normalization of
X/G .Proof.
The natural map of quotients
X/G → X/G can be viewed as the quotient map by the finitegroup
G/G and is therefore finite. The same is true for e X/G → e X/G , so by Remark 2.2(ii) it followsthat ( ν X ) G is finite if ( ν X ) G is. Hence we may assume that G is connected.The coordinate ring k [ e X ] of the normalization of X is the integral closure of k [ X ] in the function field k ( X ). Let S be the integral closure of k [ X ] G in k ( X ). Then S is finitely generated as a k -algebra [1,2.4.3], and S ⊆ k [ e X ] as k [ e X ] is integrally closed, so S ⊆ k [ e X ] G as G is connected (see the proof of [1,2.4.1]). Let Z be the affine variety corresponding to S . Then ( ν X ) G factors as e X/G α → Z β → X/G . It isclear that Z is normal (in fact, S is the integral closure of k [ X ] G in k ( X ) G , so Z is the normalization of X/G ). Now ( ν X ) G is birational and quasi-finite by Lemmas 2.9 and 2.8(ii), so α is also birational andquasi-finite. It follows from Zariski’s Main Theorem that α is an open embedding.The map β is finite by construction, so to complete the proof that ( ν X ) G is finite it is enough to showthat α is surjective. Define θ : e X → Z × X by θ = ( α ◦ π e X,G ) × ν X and let C be the closure of θ ( e X ). Wehave a commutative diagram e X π f X,G (cid:15) (cid:15) θ / / C pr (cid:15) (cid:15) e X/G α / / Z where pr is projection onto the first factor. The composition e X → C → X is finite, where the secondmap is projection onto the second factor, so θ is a finite map from e X to C ; in particular, C = θ ( e X ).Let G act on Z × X trivially on the first factor, and by the given action on the second. It is immediatethat θ is G -equivariant, so C is G -stable and we have an induced map θ G : e X/G → ( Z × X ) /G . Theimage D of θ G is π Z × X,G ( C ), and this is closed in ( Z × X ) /G as C is closed and G -stable. There is an bvious map ξ : ( Z × X ) /G → Z × X/G , and it is easily checked that ξ is an isomorphism; hence ξ ( D )is closed. Untangling the definitions, we find that α factors as e X/G θ G → ( Z × X ) /G ξ → Z × X/G τ → Z ,where τ is projection onto the first factor.Clearly ξ ( D ) is contained in the subset { ( z, e ) ∈ Z × X/G | β ( z ) = e } , which we can identify with Z via τ . It follows that α ( e X/G ) = τ ( ξ ( D )) is closed in Z . But α ( e X/G ) is a nonempty open subset of Z ,so we must have α ( e X/G ) = Z , as required.To finish the proof, we note that for any G (connected or otherwise), the variety e X/G is normal since e X is normal, and the considerations above show that ( ν X ) G : e X/G → X/G is finite. Moreover, ( ν X ) G isbirational by Lemma 2.9 since X has good dimension. The result now follows from another applicationof Zariski’s Main Theorem. (cid:3) Next we extend a result of Bardsley and Richardson [1, 2.4.2], which they prove in the special casewhen X and Y are normal and φ is dominant. It provides an extension to positive characteristic of aresult used freely in [33]. Proposition 3.10.
Let φ : X → Y be a finite G -equivariant morphism of affine G -varieties. Then φ G : X/G → Y /G is finite.Proof.
As at the start of the proof of Lemma 3.9, we can immediately reduce to the case when G isconnected, since the natural maps X/G → X/G and
Y /G → Y /G are finite. The map X cl /G → X/G is surjective, and Lemma 2.3 implies it is finite. We may also assume, therefore, that X has gooddimension. Since a morphism is finite if and only if its restriction to every irreducible component ofthe domain is finite, we can assume X is irreducible. By the proof of Lemma 2.8, φ ( X cl ) ⊆ Y cl , soafter replacing Y with φ ( X ) if necessary, we may assume by Lemma 2.3 that φ is dominant and Y isirreducible and has good dimension.The map φ : X → Y gives rise to a map e φ : e X → e Y , and e φ is finite as φ is. We have a commutativediagram e X ν X (cid:15) (cid:15) e φ / / e Y ν Y (cid:15) (cid:15) X φ / / Y where the vertical arrows are the normalization maps. Taking quotients by G , we obtain a commutativediagram e X/G ( ν X ) G (cid:15) (cid:15) e φ G / / e Y /G ( ν Y ) G (cid:15) (cid:15) X/G φ G / / Y /G
Since e φ is finite and dominant and e X and e Y are irreducible and normal, the map e φ G : e X/G → e Y /G is finite and dominant [1, 2.4.2]. Now Lemma 3.9 shows that the map ( ν Y ) G : e Y /G → Y /G is finite andso ( ν Y ) G ◦ e φ G is finite. Therefore, φ G ◦ ( ν X ) G is finite and by Remark 2.2(ii) we get that φ G is finite, asrequired. (cid:3) Proof of Theorem 1.1, Part 1: quasi-finiteness
In this section we provide the first step towards our proof of Theorem 1.1, showing that the map ψ X,H in question is quasi-finite. We are also able to retrieve other results from [33] which follow from the maintheorem, but in arbitrary characteristic. Our first result is a generalization of [2, Thm. 4.4]; see also [7,Thm. 5.4].
Proposition 4.1.
Suppose that G is a reductive group and X is an affine G -variety. Let H be a G -completely reducible subgroup of G and let x ∈ X H . Then the following are equivalent: (i) N G ( H ) · x is closed in X ; (ii) G · x is closed in X and H is G x -cr. roof. First suppose G · x is not closed. Let P ( x ) and Ω( x ) be the R-parabolic subgroup and class ofcocharacters given by Theorem 2.12. Since H ≤ G x ≤ P ( x ) is G -cr, there exists an R-Levi subgroup L of P ( x ) containing H . Since R u ( P ( x )) acts simply transitively on Ω( x ) and on the set of R-Levi subgroupsof P ( x ), there exists λ ∈ Ω( x ) with L = L λ . But then H ⊆ L λ means that λ ∈ Y ( C G ( H )) ⊆ Y ( N G ( H ));in particular, λ ( a ) · x ∈ N G ( H ) · x for all a ∈ k ∗ . Now lim a → λ ( a ) · x exists in X and is not G -conjugateto x , so it is not N G ( H )-conjugate to x , so N G ( H ) · x is not closed. This shows that if (i) holds then G · x must be closed. Therefore, in order to finish the proof, we need to show that N G ( H ) · x is closed ifand only if H is G x -cr under the assumption that G · x is closed (note that since G · x is closed, G x isreductive (Lemma 3.3(ii)), and hence it makes sense to ask whether or not H is G x -cr).To see this equivalence, let h ∈ G n for some n be a generic tuple for the subgroup H and considerthe diagonal action of G on G n × X . Then C G ( H ) = G h . Now, by Lemma 3.6, since G · x is closed in X , C G ( H ) · x is closed in X if and only if G x · h is closed in G n . The latter condition is equivalent torequiring that H is G x -cr, and since x is H -fixed and N G ( H ) is a finite extension of HC G ( H ), C G ( H ) · x is closed in X if and only if N G ( H ) · x is closed in X . This completes the proof. (cid:3) Remarks . (i). In characteristic 0, the subgroup H of G is G -cr if and only if H is reductive. In thiscase, therefore, we are just requiring that H is reductive and the second condition in part (ii) of theTheorem is then automatic. Therefore, when char( k ) = 0, we retrieve Luna’s result [33, §
3, Cor. 1].(ii). The implication (ii) implies (i) of Proposition 4.1 is not true in general without the hypothesisthat H is G x -cr, as a straightforward modification of [2, Ex. 4.6] shows. See also [6, Ex. 5.1, Ex. 5.3],noting that if A, B are commuting G -cr subgroups of G and B is not C G ( A )-cr then B is not N G ( A )-crby [6, Prop. 2.8].(iii) Suppose H is a torus in Proposition 4.1; then H is linearly reductive, so H is G -cr. Now N G ( H )is a finite extension of the Levi subgroup C G ( H ) of G , so N G ( H ) · x is closed if and only if C G ( H ) · x is closed. Moreover, H is automatically G x -cr if G x is reductive. It follows that G · x is closed if andonly if C G ( H ) · x is closed. (This is also a special case of [4, Thm. 5.4].) We use this result repeatedlyin Section 9.Some of the constructions used in the proof of the next result are based on those in [8, Sec. 3.8]. Lemma 4.3.
Suppose H is a reductive subgroup of G such that H is not G -cr. Then: (i) There exists an affine G -variety X and a point x ∈ X H such that G · x is not closed. (ii) There exists a rational G -module V and a nonzero subspace W ⊆ V H such that: (a) 0 lies in the closure of G · w for all w ∈ W ; (b) N G ( H ) · w is finite (hence closed in V ) for all w ∈ W .In particular, if N G ( H ) is reductive, then the map ψ V,H : V H /N G ( H ) → V /G is not quasi-finite.Proof.
Choose a closed embedding
G ֒ → SL m ( k ) for some m and think of H and G as closed subgroupsof SL m ( k ). Let Mat m denote the algebra of all m × m matrices. Let x = ( x , . . . , x n ) ∈ H n be a basisfor the associative subalgebra of Mat m spanned by H ; then x is a generic tuple for H (see Section 2.4).This means that if we let SL m ( k ) act on Y := (Mat m ) n by simultaneous conjugation, then G · x is notclosed. Note that since H is itself H -cr, the H -orbit of H · x is closed in Y .There is also a right action of GL n ( k ) on Y , which we denote by ∗ . Given a matrix A = ( a ij ) ∈ GL n ( k )and an element y = ( y , . . . , y n ) ∈ Y , we can set y ∗ A = n X i =1 a i y i , . . . , n X i =1 a in y i ! . This is the action obtained by thinking of the tuple y as a row vector of length n and letting the n × n matrix A act on the right in the obvious way. Note that the SL m ( k )- and GL n ( k )-actions commute.Given any h ∈ H , since x is a basis for the associative algebra generated by H , we have that h · x isalso a basis for this algebra, and hence there exists a unique A ( h ) ∈ GL n ( k ) such that h · x = x ∗ A ( h ).Note also that( h h ) · x = h · ( h · x ) = h · ( x ∗ A ( h )) = ( h · x ) ∗ A ( h ) = x ∗ ( A ( h ) A ( h )) , and hence the map A : H → GL n ( k ) is a group homomorphism. This map is in fact a rational repre-sentation of H since it arises from the morphic action of H on the vector space spanned by the entriesof x . Let K denote the image of H in GL n ( k ); then K is a reductive group and x ∗ K = H · x is closed.Moreover, since the elements of the tuple x are linearly independent, the stabilizer of x in K is trivial.Hence x is a stable point for the action of K on Y . Now let X = Y /K and set x := π Y,K ( x ). Since the L m ( k )- and GL n ( k )-actions on Y commute, we obtain an action of SL m ( k ) on X . It is immediate that x ∈ X H .We know that G · x is not closed in Y , so there exists a cocharacter λ ∈ Y ( G ) such that lim a → λ ( a ) · x = y exists and is not G -conjugate to x . Since π Y,K is G -equivariant, it is easy to see that lim a → λ ( a ) · x = π Y,K ( y ) (and in particular this limit exists). Suppose π Y,K ( y ) is G -conjugate to x . Then there exists g ∈ G such that g · π Y,K ( y ) = π Y,K ( g · y ) = x , so g · y ∈ π − Y,K ( x ) = π − Y,K ( π Y,K ( x )). But x is a stablepoint for K , so π − Y,K ( π Y,K ( x )) is precisely K · x , which coincides with H · x by construction. Hence g · y = h · x for some h ∈ H and we see that y and x are G -conjugate, which is a contradiction. Hence π ( y ) and x are not conjugate, and the G -orbit of x ∈ X H is not closed, which proves (i).To prove (ii), let S denote the unique closed G -orbit in the closure of G · x . Then, following [28,Lemma 1.1(b)], we can find a rational G -module V with a G -equivariant morphism φ : X → V suchthat φ − (0) = S . Since G · x is not closed, it does not meet S , and hence v := φ ( x ) = 0. However,by Theorem 2.12, there exists µ ∈ Y ( G ) such that lim a → µ ( a ) · x ∈ S , and since the morphism φ is G -equivariant, we have that { } is the unique closed G -orbit in the closure of G · v . Note also that v is H -fixed since x is. Now the tuple x consists of elements of H , so is C G ( H )-fixed, and hence x = π Y,K ( x ) is also C G ( H )-fixed, which means that x is actually HC G ( H )-fixed. Since H is reductive, N G ( H ) = H C G ( H ) , so x is N G ( H ) -fixed and hence the N G ( H )-orbit of x is finite. This in turnimplies that the N G ( H )-orbit of v is finite, and hence closed in V . Finally, let W ⊆ V H be the one-dimensional subspace of V spanned by v . Then for all w ∈ W , N G ( H ) · w is finite, hence closed,and 0 is in the closure of G · w , so we have parts (a) and (b) of (ii). For the second statement, if N G ( H ) is reductive—so that it definitely makes sense to talk about the quotient V H /N G ( H )—then theimage of W in V H /N G ( H ) is still infinite, but every element of this infinite set is mapped to the pointcorresponding to 0 in V /G under the natural morphism V H /N G ( H ) → V /G , so this morphism cannotbe quasi-finite. (cid:3)
With this result in hand, we can provide the first step towards the proof of Theorem 1.1 by showingthat the morphism ψ X,H is quasi-finite.
Theorem 4.4.
Suppose H is a reductive subgroup of G . The following conditions on H are equivalent: (i) N G ( H ) is reductive and for every affine G -variety X , the natural morphism ψ X,H : X H /N G ( H ) → X/G is quasi-finite; (ii) H is G -cr.Proof. Suppose H is not G -cr. Then either N G ( H ) is not reductive, in which case the first part ofcondition (i) fails, or else N G ( H ) is reductive but the second part of condition (i) fails by Lemma 4.3(ii).Hence (i) implies (ii).Conversely, suppose H is G -cr, and let X be any affine G -variety. Since H is G -cr, and hence H isreductive, we have N G ( H ) = H C G ( H ) . That N G ( H ) is reductive is shown in [5, Prop. 3.12], andhence it always makes sense to take the quotient X H /N G ( H ).Suppose x ∈ X H . We first claim that the unique closed G -orbit S in G · x meets X H . Indeed, either G · x is already closed, in which case S = G · x , or we can find the optimal parabolic P ( x ) and optimalclass Ω( x ) as given in Kempf’s Theorem 2.12. Since H ≤ G x ≤ P ( x ) and H is G -cr, there is a Levisubgroup L of P ( x ) containing H . Since the unipotent radical acts simply transitively on Ω( x ) and onthe set of Levi subgroups of P ( x ), there is precisely one element λ ∈ Ω( x ) with L = L λ , and this choiceof λ commutes with H . But then y := lim a → λ ( a ) · x ∈ S ∩ X H , which proves the claim.Now any point of X/G has the form π X,G ( x ), where G · x is closed in X . So let x ∈ X such that G · x is closed. For any y ∈ π − X,G ( π X,G ( x )) ∩ X H , G · x is the unique closed G -orbit in G · y . Hence, if π − X,G ( π X,G ( x )) ∩ X H is nonempty, G · x must meet X H , by the claim in the previous paragraph. It followsfrom the definitions that π − X H ,N G ( H ) ( ψ − X,H ( π X,G ( x ))) = π − X,G ( π X,G ( x )) ∩ X H , so to show that ψ X,H isquasi-finite, we need to show that for each such x there are only finitely many closed N G ( H )-orbits in π − X,G ( π X,G ( x )) ∩ X H . But any y ∈ X H with a closed N G ( H )-orbit has a closed G -orbit, by Proposition4.1, and hence any y ∈ π − X,G ( π X,G ( x )) ∩ X H with a closed N G ( H )-orbit is already G -conjugate to x . Sowe must show that there are only finitely many closed N G ( H )-orbits in G · x ∩ X H .Fix x ∈ X H with G · x closed, and recall that G x is reductive since G · x is closed. Let y ∈ G · x ∩ X H ,and write y = g · x for some g ∈ G . Since G · y is closed, Proposition 4.1 says that N G ( H ) · y is closedif and only if H is G y -cr, which is the case if and only if g − Hg is G x -cr. Suppose g − Hg and H are G x -conjugate: say H = g − ( g − Hg ) g for some g ∈ G x . Then gg ∈ N G ( H ) and y = g · x = ( gg ) · x , o we see that x and y are N G ( H )-conjugate. Conversely, suppose x and y are N G ( H )-conjugate: say y = m · x for some m ∈ N G ( H ). Then m − g ∈ G x and m − g ( g − Hg ) g − m = H , so g − Hg and H are G x -conjugate. Hence the distinct closed N G ( H )-orbits in G · x ∩ X H correspond to the distinct G x -conjugacy classes of G x -cr subgroups of the form g − Hg inside G x . It is therefore enough to showthat there are only finitely many such conjugacy classes.Let h ∈ H n be a generic tuple for H in G x for some n and let g ∈ G such that g − Hg is a G x -crsubgroup of G x . Then g − · h is a generic tuple for g − Hg . Since g − Hg is both G -cr and G x -cr, the G - and G x -orbits of h in G n are both closed. It follows from [35, Thm. 1.1] that the natural map ofquotients G nx /G x → G n /G is finite, and hence there are only finitely many closed G x -orbits containedin G · h ∩ G nx . This proves the result. (cid:3) Remark . Note that if G x = H and G · x is closed then the argument in the proof above showsthat there is precisely one closed N G ( H )-orbit inside G · x ∩ X H (namely, N G ( H ) · x ), and therefore ψ − X,H ( π X,G ( x )) is a singleton. We will use this observation in Sections 6 and 7.The third paragraph of the proof above shows that for any x ∈ X H , the unique closed orbit containedin G · x also meets X H . This allows us to prove the following: Lemma 4.6.
The map ψ X,H : X H /N G ( H ) → X/G of Theorem 1.1 has closed image if and only if forall x ∈ G · X H such that G · x is closed, x ∈ G · X H .Proof. Since G · X H is closed and G -stable, we may replace X with G · X H ; then saying ψ X,H has closedimage is the same as saying that ψ X,H is surjective. But this is equivalent to saying that the fibre aboveevery point of
X/G meets X H . Since each fibre contains a unique closed orbit, the observation beforethe Lemma gives the result. (cid:3) Now we extend Luna’s result [33, Cor. 3] to positive characteristic.
Proposition 4.7.
Suppose H is a reductive subgroup of G . The following conditions on H are equivalent: (i) for every affine G -variety X , every G -orbit in X that meets X H is closed; (ii) H is G -cr and N G ( H ) /H is a finite group.Proof. Suppose (i) holds. Then H must be G -cr, by Lemma 4.3. Since H is reductive, N G ( H ) = H C G ( H ) . Let x ∈ C G ( H ) and let G act on itself by conjugation. We have x ∈ C G ( H ) = G H , so the G -orbit of x (i.e., the conjugacy class of x ) must be closed in G . As x belongs to G , it follows from [55,Cor. 3.6] that x is a semisimple element of G . Since C G ( H ) consists entirely of semisimple elements, itmust be a torus [10, Cor. 11.5(1)]. Hence N G ( H ) = H C G ( H ) is a reductive group and ( N G ( H ) /H ) is a torus.Now suppose, for contradiction, that N G ( H ) /H is infinite. Then there exists a one-dimensionalsubtorus S of C G ( H ) not contained in H . To ease notation, let Z = HS and note that Z is reductive.Since H is normal in Z and Z/H ∼ = S/ ( S ∩ H ) is a one-dimensional torus, we have a multiplicativecharacter χ : Z → k ∗ with kernel H ; let V denote the corresponding 1-dimensional Z -module. Set Y = G × V , let Z act on Y via z · ( g, v ) := ( gz − , χ ( z ) v ), and let G act by left multiplication on thefirst factor and trivially on the second factor. Now let X = Y /Z ; this is a special case of a constructiondescribed in [32, I.3]. Since Z is reductive and Y is affine, X is affine, and since Z acts freely on Y , the fibres of π Y,Z are precisely the Z -orbits in Y . Moreover, since the G - and Z -actions on Y commute, X is naturally a G -variety. Let 0 = v ∈ V and choose a cocharacter λ of Z such that m := −h λ, χ i >
0. Then λ ( a ) · π Y,Z (1 , v ) = π Y,Z (1 , χ ( λ ( a − )) v ) = π Y,Z (1 , a m v ) for all a ∈ k ∗ , solim a → λ ( a ) · π Y,Z (1 , v ) = π Y,Z (1 , G · π Y,Z (1 , v ), so G · π Y,Z (1 , v ) is not closed. However, π Y,Z (1 , v )is H -fixed, and we have our contradiction. Hence N G ( H ) /H is finite. This completes the proof that (i)implies (ii).Conversely, suppose (ii) holds and X is any affine G -variety. Let x ∈ X H , so that H ≤ G x . Since N G ( H ) /H is finite, N G ( H ) · x is a finite union of H -orbits. But N G ( H ) · x ⊆ X H , so each of these H -orbits is a singleton and N G ( H ) · x is finite, and therefore closed in X . Now we can apply Proposition4.1 to deduce that the G -orbit of x is also closed, which gives (i). (cid:3) Proof of Theorem 1.1, Part 2: surjectivity
In this section, we prove the following:
Theorem 5.1.
Let X be an affine G -variety and let H be a G -cr subgroup of G . Then the map ψ X,H : X H /N G ( H ) → X/G has closed image. he proof of Theorem 5.1 in positive characteristic requires some preparation. Before we begin, wenote that if char( k ) = 0 then we can give a much quicker proof using the machinery of ´etale slices, asfollows. Let x ∈ G · X H such that G · x is closed. Then there is an ´etale slice through x for the G -action[32, III.1]. By Proposition 3.1, there is an open G -stable neighbourhood O of x such that G y is conjugateto a subgroup of G x for all y ∈ O . Since O meets G · X H , H must be conjugate to a subgroup of G x .Hence x ∈ G · X H , and we are done by Lemma 4.6.We need some material on weighted projective varieties and their quotients by reductive groups (cf.[44, Chapter 3, § V be a G -module equipped with an action of k ∗ which commutes with theaction of G . Suppose the weights of k ∗ on V are all positive, so that lim c → c · v = 0 for every v ∈ V ,where the c ∈ k ∗ . The action of k ∗ decomposes V into weight spaces, and this in turn gives a grading bynon-negative integers of the coordinate ring k [ V ]. Let k [ V ] i denote the i th -graded piece of k [ V ]. We saythat f ∈ k [ V ] is homogeneous if f ∈ k [ V ] i for some i ; in this case we write deg( f ) = i (so deg( f ) is theweighted degree rather than the usual degree of a polynomial). The action of k ∗ can be diagonalised, sowe can choose a basis { v , . . . , v n } for V consisting of weight vectors. Then the corresponding elements X , . . . , X n of the dual V ∗ are weight vectors and we can write k [ V ] = k [ X , . . . , X n ]; we set d i = deg( X i )for each i .Set W ( V ) = Proj( k [ V ]); we call this the weighted projectivization of V according to the k ∗ -action[18]. Then W ( V ) is a projective variety and we may identify the points of W ( V ) with the equivalenceclasses of V \ { } under the equivalence relation ∼ , where v ∼ w if and only if v = c · w for some c ∈ k ∗ . If the weights of the k ∗ -action on V are all 1—that is, if the action of k ∗ on V is by ordinaryscalar multiplication—then W ( V ) is just the ordinary projective space P ( V ) associated to V , but inSection 6 we will need to consider the general weighted case. One can show that the canonical projection ξ V : V \{ } → W ( V ) is a good quotient. If f ∈ k [ V ] is homogeneous and deg( f ) ≥ W ( V ) f = { ξ V ( v ) | v ∈ V, f ( v ) = 0 } ; then W ( V ) f is an open affine subset of W ( V ), with coordinate ring( k [ V ] f ) (the zero-graded part of the localisation k [ V ] f ).Since the G - and k ∗ -actions commute, the ring k [ V ] G of invariants also inherits a grading by non-negative integers: if f ∈ k [ V ] G and f = f + · · · + f r is a decomposition with f i ∈ k [ V ] i for each i , then f i ∈ ( k [ V ] G ) i for each i . It is easily checked that the action of G on V descends to give an action of G on W ( V ). We say that x ∈ W ( V ) is a semistable point (or G -semistable point ) if x ∈ W ( V ) f for somehomogeneous f ∈ k [ V ] G such that deg( f ) ≥
1; otherwise we say that x is unstable (or G -unstable ). Wedefine W ( V ) ss ,G to be the set of G -semistable points of W ( V ); this is an open subset of W ( V ).Let Y = Proj( k [ V ] G ). Then Y is a projective variety and the inclusion of k [ V ] G in k [ V ] gives riseto a map η V,G : W ( V ) ss ,G → Y . It follows from the proof of [44, Thm. 3.14] that Y is a good quotientof W ( V ) ss ,G in the sense of [44, Chapter 3, §
4, p57] (the argument given in loc. cit. is only for theordinary projective variety P ( V ), but it is clear that it holds for the weighted case as well). We set W ( V ) ss ,G /G := Y . Moreover, if f ∈ k [ V ] G is homogeneous and deg( f ) ≥ Y f := η V,G ( W ( V ) f ) isan open affine subvariety of Y , with coordinate ring (( k [ V ] G ) f ) , and the induced map of affine varietiesfrom W ( V ) f to Y f is a good quotient.We have an analogous notion of semistable points in the affine variety V . We say that v ∈ V semistable (or G -semistable ) if f ( v ) = 0 for some homogeneous f ∈ k [ V ] G such that deg( f ) ≥
1, and we define V ss ,G to be the set of semistable points; note that V ss ,G = ξ − V ( W ( V ) ss ,G ). If v is not stable then we say that v is unstable (or G -unstable ). Since the homogeneous elements of k [ V ] G generate k [ V ] G , v is unstable ifand only if π V,G ( v ) = π V,G (0). By the Hilbert-Mumford Theorem, this is the case if and only if thereexists λ ∈ Y ( G ) such that lim a → λ ( a ) · v = 0. We denote the composition V ss ,G → W ( V ) ss ,G η V,G −→ Y by ν V,G .Now suppose K is a reductive subgroup of G and X is a closed ( K × k ∗ )-stable subvariety of V (so thatin particular 0 ∈ X ). Then the vanishing ideal for X in k [ V ] is homogeneous with respect to our fixed k ∗ -grading, so k [ X ] inherits a grading. The constructions above still go through replacing V and G with X and K . We have projective varieties W ( X ) := Proj( k [ X ]) and W ( X ) ss ,K /K := Z := Proj( k [ X ] K ),where W ( X ) ss ,K is defined analogously to above; the map W ( X ) ss ,K → Z is a good quotient. Note thatthe proof of [44, Thm. 3.14] still goes through: all one needs is that k [ X ] is graded and the G -actionpreserves the grading. Since W ( V ) and W ( X ) are categorical quotients of V \{ } and X \{ } respectively,the inclusion of X in V gives rise to a map from W ( X ) to W ( V ).It is clear from the characterisation of semistable points in terms of the Hilbert-Mumford Theoremthat X ∩ V ss ,G ⊆ X ss ,K . Suppose X ss ,K ⊆ V ss ,G ; then X ss ,K = X ∩ V ss ,G . Since Y and Z are categoricalquotients of W ( V ) ss ,G and W ( X ) ss ,K , respectively, the inclusion of W ( X ) ss ,K in W ( V ) ss ,G gives rise toa map φ : Z → Y . Now we come to the point: because Y and Z are projective, the image of φ is closed. e can now state and prove the main result of this section. Proposition 5.2.
Let V be a G -module equipped with a k ∗ -action as above. Let K be a reductivesubgroup of G , let X be a closed ( K × k ∗ ) -stable subset of V and suppose X ss ,K ⊆ V ss ,G . Then thenatural morphism of quotients X/K → V /G has closed image (i.e., π V,G ( X ) is closed in V /G ).Proof.
For the purposes of the proof, we need to replace G with a slightly larger group to take intoaccount possible effects of passing to the weighted projectivisation. Without loss, we may assume that G is a subgroup of GL( V ). Let R = k [ V ] G and let f , . . . , f r be homogeneous generators for R . Let m bethe lowest common multiple of the degrees deg( f ) , . . . , deg( f r ) and write m = p α m ′ for some m ′ coprimeto p . Let F be the finite group of m ′ th roots of unity, regarded as a subgroup of k ∗ (equipped with itsgiven action on V ). Now set Γ = F G . Then Γ inherits an action on V from the commuting actions of G and k ∗ , and V ss , Γ = V ss ,G because Γ = G . Further, F acts on the quotient V /G and the quotient map π V/G,F : V /G → ( V /G ) /F = V /
Γ is a geometric quotient. The subset X of V is k ∗ -stable, and hence F -stable, so π V,G ( X ) is a π V/G,F -saturated subset of
V /G – that is, π − V/G,F ( π V/G,F ( π V,G ( X ))) = π V,G ( X ).Hence, to show the result claimed, we may replace G with Γ and show that π V, Γ ( X ) is closed in V /
Γ.Now let S = k [ V ] Γ ⊆ k [ V ] G . A homogeneous f ∈ R belongs to S if and only if deg( f ) is divisible by m ′ (since then the action of F is killed by the degree).To show that π V, Γ ( X ) is closed in V /
Γ, it is enough to show that for every x ∈ Γ · X with closedΓ-orbit, there exists an x ′ ∈ X with π V, Γ ( x ′ ) = π V, Γ ( x ) (cf. Lemma 4.6). If x ∈ Γ · X is unstable andΓ · x is closed, then x must actually be 0, so x ∈ X also. Therefore, we may assume that we have x ∈ V ss , Γ ∩ Γ · X such that Γ · x is closed (in V ). By the discussion before the proposition, we have amorphism φ : W ( X ) ss ,K /K → W ( V ) ss , Γ / Γ with closed image C , say. Note that since we are assuming X ss ,K = V ss , Γ ∩ X , we have Γ · X ss ,K = V ss , Γ ∩ Γ · X , and this set is dense in V ss , Γ ∩ Γ · X . The composition V ss , Γ ξ V −→ W ( V ) ss , Γ η V, Γ −→ W ( V ) ss , Γ / Γ takes Γ · X ss ,K into C , and since C is closed that means that thecomposition in fact takes all of V ss , Γ ∩ Γ · X into C . Therefore, we can find z in W ( X ) ss ,K /K with φ ( z ) = η V, Γ ( ξ V ( x )). Tracing back through the definitions, we see that φ ( z ) = η V, Γ ( ξ V ( y )) for some y ∈ X ss ,K . It follows that η V, Γ ( ξ V ( x )) = η V, Γ ( ξ V ( y )); we claim that in fact π V, Γ ( x ) = π V, Γ ( c · y ) for some c ∈ k ∗ . Note that suffices to finish the proof, since in particular y ∈ X , so setting x ′ = c · y ∈ X gives uswhat we want.Both points x and y lie in V ss , Γ , so there are homogeneous generators f i , f j ∈ R for which f i ( x ) = 0and f j ( y ) = 0. By definition of m , there are m i , m j ∈ N such that f m i i and f m j j both have degree m .Taking a suitable linear combination of f m i i and f m j j , we can therefore find a homogeneous f ∈ R ofdegree m for which f ( x ) = 0 = f ( y ). Now we can choose c ∈ k ∗ such that f ( x ) = f ( c · y ).Let f ′ ∈ S be non-constant and homogeneous; as previously observed, f ′ ∈ S means that deg( f ′ ) = rm ′ for some r ∈ N . Then ( f ′ ) p α has degree rm , so ( f ′ ) pα f r has degree 0 in the localization R f . Further,since x and c · y have the same image in ( W ( V ) ss , Γ / Γ) f , we have ( f ′ ) pα f r ( x ) = ( f ′ ) pα f r ( c · y ). Since f ( x ) = f ( c · y ) = 0, this in turn implies that ( f ′ ) p α ( x ) = f ′ p α ( c · y ), and hence f ′ ( x ) = f ′ ( c · y ). Since S is generated by homogeneous elements, we see that π V, Γ ( x ) = π V, Γ ( c · y ), as required. This finishes theproof. (cid:3) Proof of Theorem 5.1.
We can choose a G -equivariant closed embedding of X in a G -module V . Let v ∈ V H such that v is G -unstable. By the argument in the proof of Proposition 4.1, there exists λ ∈ Y ( N G ( H )) such that lim a → λ ( a ) · v = 0. This shows that ( V H ) ss ,N G ( H ) ⊆ V ss ,G . The G -actioncommutes with the natural k ∗ -action by scalars, and this preserves the subspace V H also, so Proposition5.2 implies that the map ψ V,H : V H /N G ( H ) → V /G has closed image.Now G · X H ∩ V H ⊆ X ∩ V H = X H , so G · X H ∩ V H = X H . Let x ∈ G · X H such that G · x isclosed. Then, since x ∈ G · V H and ψ V,H has closed image, Lemma 4.6 implies x ∈ G · V H , so we canwrite x = g · v for some v ∈ V H . But then v ∈ G · X H ∩ V H = X H , so x ∈ G · X H and we are done byLemma 4.6. (cid:3) Remark . For the proof of Theorem 5.1, we only need to apply Proposition 5.2 when the k ∗ -action isthe standard action by scalars, so the weighted projectivization is the usual projectivization in this case.However, we do need Proposition 5.2 in this more general set-up to complete the proof of Theorem 1.1in the next section. Example . Let G act on X := G by conjugation and let H be a maximal torus of G . Assume G is connected. Then X H = H . Since the closed orbits in X are precisely the semisimple conjugacy lasses [55], the map ψ X,H : X H /N G ( H ) → X/G is surjective—in fact, it is well known that ψ X,H is anisomorphism (cf. Section 7). Note, however, that although G · X H is dense in X , not every element of X belongs to G · H (just take x ∈ X not semisimple).6. Proof of Theorem 1.1, Part 3: finiteness
We now complete the proof of Theorem 1.1. The implication (ii) = ⇒ (i) follows from Theorem 4.4, soit remains to show that if H is G -completely reducible then the morphism ψ X,H is finite. By Lemma 2.3,we can replace X with a larger affine G -variety, hence without loss we can assume that X is a G -module.Let G be the subgroup of G generated by G and H . The inclusion of X H in X gives rise to amorphism ψ X,H : X H /N G ( H ) → X/G . We have a commutative diagram(6.1) X H /N G ( H ) (cid:15) (cid:15) ψ X,H / / X/G (cid:15) (cid:15) X H /N G ( H ) ψ X,H / / X/G where the vertical arrows are the obvious maps. We may identify
X/G with the quotient of
X/G by thefinite group G/G , so the map X/G → X/G is finite. This map factorizes as
X/G → X/G → X/G ,so the map
X/G → X/G is finite by Remark 2.2(ii). Likewise, the map X H /N G ( H ) → X H /N G ( H )is finite. Hence both of the vertical maps in (6.1) are finite and surjective. Now using Remark 2.2(ii)we see it is enough to show that ψ X,H is finite. So it is enough to prove that ψ X,H is finite under theassumption that G = G .Let Y and U ⊆ Y H be as in Lemma 2.11 and set V = X ⊕ Y . We have a G -equivariant closedembedding of X in V given by x ( x, W = G · V H ; then W H = V H . Note that G · V H = G · V H by our assumption that G = G , so W is irreducible. By Lemma 2.3 again, it is enough to prove that ψ W,H is finite.The subset X H × U of X H ⊕ Y H = V H = W H is open and dense, and G w = H for w ∈ X H × U . Nextwe claim that W has good dimension (for the G -action). To see this, let y ∈ U and set w = (0 , y ).Then G · w is a G -orbit of maximal dimension in W , and G · w is closed (as G · y is, by Lemma 2.11),so w is a stable point of W for the G -action. The claim now follows from Remark 2.6. By a similarargument, W H has good dimension for the N G ( H )-action. Now since the stable points form an opensubset, we can conclude that G · w and N G ( H ) · w are closed for generic w ∈ W H , and it follows fromRemark 4.5 that generic fibres of ψ W,H are singletons.Now consider the normalization f W of W . Since the normalization map ν W : f W → W is birational, f W contains an open dense subvariety e O such that the map e O → W is an isomorphism onto its image O ,and O is open in W . We can take e O and O to be G -stable, so the latter meets G · W H . Now G · W H is constructible and dense in W , so it contains a nonempty open subset of W . Hence, by adjusting O and e O if necessary, we can assume that O ⊆ G · W H and O is N G ( H )-stable. Since ν W is G -equivariant,we get an isomorphism from e O H onto O H . Let C be the closure in f W of e O H ; then C ⊆ f W H and C is N G ( H )-stable. Further, since the open subset e O H in C is isomorphic to the open subset O H in theirreducible set W H , C is irreducible; it follows from Zariski’s Main Theorem that C is isomorphic to W H . Hence C ∼ = W H = V H carries a vector space structure, and we can identify a point 0 C ∈ C corresponding to the zero 0 V ; we have ν W (0 C ) = 0 V by construction. Furthermore, the action of k ∗ on W by scalar multiplication lifts to an action of k ∗ on f W which preserves the closed subset C .We want to apply Proposition 5.2 to deduce that the map C/N G ( H ) → f W /G has closed image (notethat we cannot use Theorem 5.1 directly because C might be properly contained in f W H ). In order todo this, we choose a ( G × k ∗ )-equivariant embedding i of f W in a vector space M such that 0 C maps tothe zero 0 M ∈ M . (For instance, choose f , . . . , f s ∈ k [ f W ] for some s such that the f i generate k [ f W ]as a k -algebra and f (0 C ) = · · · = f s (0 C ) = 0; we can take M to be the dual of N , where N is a( G × k ∗ )-stable subspace of k [ f W ] containing all the f i .) Replacing M with the subspace spanned by i ( f W ), we can assume that i ( f W ) spans M .Let λ : k ∗ → k ∗ be the identity cocharacter of k ∗ . Now { V } is the unique closed k ∗ -orbit in W , andeach element of W is destabilized to 0 V by λ . It follows from Lemma 2.8(i) that { C } is the unique closed k ∗ -orbit in f W . Let 0 C = e w ∈ f W . The Hilbert-Mumford Theorem implies that lim a → λ ( a ) · e w = 0 C or im a → ( − λ )( a ) · e w = 0 C . In particular, k ∗ does not fix e w , so k ∗ does not fix ν W ( e w ), so ν W ( e w ) = 0 V .Suppose lim a → ( − λ )( a ) · e w = 0 C . Then lim a → ( − λ )( a ) · ν W ( e w ) = ν W (0 C ) = 0 V . But this is impossiblebecause lim a → λ ( a ) · ν W ( e w ) = 0 V and ν W ( e w ) = 0 V . We deduce that lim a → λ ( a ) · e w = 0 C . Hence,we can conclude that k ∗ acts on M with positive weights. Further, C is a closed ( N G ( H ) × k ∗ )-stablesubset of M . By the same argument as in the proof of Proposition 4.1, if c ∈ C is G -unstable, thensince c is H -fixed and H is G -cr, c is also N G ( H )-unstable. Hence C ss ,N G ( H ) ⊆ M ss ,G . Thus we can nowapply Proposition 5.2 to deduce that C/N G ( H ) has closed image in M/G . Since i G : f W /G → M/G isinjective (Lemma 2.3), we deduce that
C/N G ( H ) has closed image in f W /G , as we wanted. This allowsus to draw the following commutative diagram:
C/N G ( H ) (cid:15) (cid:15) ψ f W,H / / f W /G ( ν W ) G (cid:15) (cid:15) W H /N G ( H ) ψ W,H / / W/G where by abuse of notation we denote the restriction of ψ f W ,H to C/N G ( H ) by the same symbol. Theleftmost vertical arrow is the isomorphism induced by the isomorphism C ∼ = W H above. The othervertical map is finite (Proposition 3.10) and birational (Lemma 2.9; recall that W has good dimension).By Theorem 5.1, ψ W,H has closed image, and we have just argued that ψ f W ,H ( C/N G ( H )) is closed.But W = G · W H , so ψ W,H is surjective, and it follows that ψ f W ,H ( C/N G ( H )) = f W /G . Since ψ W,H isquasi-finite (Theorem 4.4) and has singletons as generic fibres, the same is true of ψ f W ,H . As f W /G isnormal, it follows from Proposition 3.8 that ψ f W ,H is finite and bijective. This implies that ( ν W ) G ◦ ψ f W ,H is finite. Since the leftmost vertical arrow is an isomorphism, we have that ψ W,H is finite, as required.This completes the proof of Theorem 1.1.7.
Separability of ψ X,H
We now consider the question of when ψ X,H is an isomorphism, or close to being one. Before we stateour result, we need some terminology.
Definition 7.1.
Let H be a subgroup of G . We say that H is a principal stabilizer for the G -variety X if there exists a nonempty open subset U of X such that G x is G -conjugate to H for all x ∈ U . Wesay that H is a principal connected stabilizer for the G -variety X if H is connected and there exists anonempty open subset U of X such that G x is G -conjugate to H for all x ∈ U . It is immediate thatif G permutes the irreducible components of X transitively then a principal stabilizer (resp., principalconnected stabilizer) is unique up to conjugacy, if one exists. In characteristic 0, principal stabilizers exist under mild hypotheses: for instance, if X is smooth [50,Prop. 5.3] or if X has good dimension [34, Lem. 3.4]. For a counterexample in positive characteristic,see Example 3.2. Theorem 7.2.
Let X be an affine G -variety. Suppose that: (a) H is a principal stabilizer for X cl ; (b) H is G -cr; (c) X/G and X H /N G ( H ) are irreducible; and (d) X/G is normal. Then ψ X,H is finite andbijective. In particular, if ψ X,H is separable then it is an isomorphism.
Observe that this result extends a theorem of Luna and Richardson [34, Thm. 4.2] to positive charac-teristic; note that in characteristic 0, a reductive group H is automatically G -cr, ψ X,H is automaticallyseparable and principal stabilizers exist, as noted above.
Proof.
By Theorem 1.1, ψ X,H is finite, so its fibres are finite. To prove the first assertion of the theoremit is enough, therefore, by Proposition 3.8 to show that ψ X,H is surjective and generic fibres of ψ X,H are singletons. By hypothesis, G · X H contains a nonempty open subset of X cl . The assumptionthat X/G is irreducible implies that the action of G is transitive on the irreducible components of X cl so we can conclude that π X,G ( X H ) = π X,G ( G · X H ) contains a nonempty open subset of X/G .As
X/G is irreducible, ψ X,H ( X H /N G ( H )) = X/G . If x is a stable point of X cl and G x = H then ψ − X,H ( ψ X,H ( π X H ,N G ( H ) ( x ))) is a singleton, by Remark 4.5. This proves the first assertion as the set ofconjugates of such x is open in X cl . If ψ X,H is separable then the second assertion follows from Zariski’sMain Theorem, as
X/G is normal. (cid:3) emark . The assertion of Theorem 7.2 also holds by a similar argument if we replace the hypothesisthat H is a principal stabilizer for X cl with the hypothesis that H is a principal connected stabilizer for X cl .Next we study the separability condition. To simplify the arguments below, we consider only the casewhen X has good dimension for the G -action. Lemma 7.4.
Suppose an affine G -variety X has good dimension and hypotheses (a)–(c) of Theorem 7.2hold. Then ψ X,H is separable if and only if for generic x ∈ X H , T x ( G · x ) ∩ T x X H = T x ( N G ( H ) · x ) .Proof. Clearly T x ( G · x ) ∩ T x X H ⊇ T x ( N G ( H ) · x ), so the content here is in the reverse inclusion. Firstwe claim that X H has good dimension for the N G ( H )-action. To see this, observe that G · X H = X by the surjectivity assertion of Theorem 7.2 (which does not depend on hypothesis (d)), so every closed G -orbit in X meets X H by Lemma 4.6. As H is a principal stabilizer for X , we must have G x = H for generic x ∈ X H , and it follows from Proposition 4.1 that generic N G ( H )-orbits in X H are closed, asrequired. We now see from Remark 2.6 that(7.5) π − X,G ( π X,G ( x )) = G · x and π − X H ,N G ( H ) ( π X H ,N G ( H ) ( x )) = N G ( H ) · x for generic x ∈ X H . Now π X,G and π X H ,N G ( H ) are separable (Lemma 2.7), and it follows from this andfrom Eqn. (7.5) that for generic x ∈ X H , d x π X,G is surjective at x with kernel T x ( G · x ) and d x π X H ,N G ( H ) is surjective at x with kernel T x ( N G ( H ) · x ).The map ψ X,H is surjective and finite (by Theorem 1.1), so it is separable if and only its derivative isan isomorphism for generic points in X H /N G ( H ). The result now follows from the argument above. (cid:3) Recall that a pair (
G, H ) of reductive groups with H ≤ G is called a reductive pair if h = Lie( H )splits off as a direct H -module summand of g = Lie( G ), where H acts via the adjoint action of G on g ,and a subgroup A ≤ G is called separable in G ifLie( C G ( A )) = c g ( A ) := { X ∈ g | Ad G ( a )( X ) = X for all a ∈ A } . Proposition 7.6.
Suppose an affine G -variety X has good dimension and hypotheses (a)–(c) of Theo-rem 7.2 hold. Suppose one of the following holds: (i) there exists x ∈ X such that G x = H and there is an ´etale slice through x for the G -action; (ii) H is separable in G , ( G, H ) is a reductive pair and there exists x ∈ X such that G x = H and G · x is separable.Then ψ X,H is separable.Proof.
By the argument of Theorem 7.2, ψ X,H is dominant. Suppose first that (i) holds. Let x ∈ X with G x = H and let S be an ´etale slice through x for the G -action. By the definition of ´etale slices andthe proof of [1, Prop. 8.6], there exists a G -stable open neigbourhood U of x in X such that G y ≤ G x for all y ∈ S ∩ U and the obvious maps G × ( S ∩ U ) → X and ( S ∩ U ) /H → X/G are ´etale. As H is aprincipal stabilizer for X , we can assume after replacing U with a smaller open set that G y is conjugateto H for all y ∈ S ∩ U . We have G x = H by hypothesis, so it follows that G y = H for all y ∈ S ∩ U .As the set of stable points of X is G -stable, open and nonempty and the set of smooth points of X/G is open and nonempty, there is a nonempty G -stable open subset U of U such that G · y is closed and π X,G ( y ) is a smooth point of X/G for all y ∈ U .Since G · ( S ∩ U ) is open and S ∩ U ⊆ X H , G · ( S ∩ U ) contains a nonempty open subset of X H . Let y ′ ∈ X H ∩ G · ( S ∩ U ): say, y ′ = g · y for some y ∈ S ∩ U , g ∈ G . Then G y = H and G y ′ is G -conjugate to H ; but y ′ ∈ X H , so G y ′ = H . It follows that g ∈ N G ( H ). We deduce that X H ∩ G · ( S ∩ U ) = N G ( H ) · ( S ∩ U ). So π X H ,N G ( H ) ( S ∩ U ) = π X H ,N G ( H ) ( N G ( H ) · ( S ∩ U )) containsa nonempty open subset of X H /N G ( H ).So pick y ∈ S ∩ U such that π X H ,N G ( H ) ( y ) is a smooth point of X H /N G ( H ). The map ( S ∩ U ) /H → X/G is ´etale, so its derivative is an isomorphism everywhere. Hence the derivative of the map X H → X/G induced by π X,G is surjective at y . This in turn implies that the derivative of ψ X,H is surjective at π X H ,N G ( H ) ( y ). But π X H ,N G ( H ) ( y ) and π X,G ( y ) are smooth points by construction, so ψ X,H is separable.Now suppose that (ii) holds. We argue along the lines of the proof of [48, Thm. A]. Let d be an H -module complement to h in g . Let X = { x ∈ X | G x = H and G · x is closed and separable } . Let x ∈ X . Then the orbit map κ x : G → G · x gives an isomorphism φ : G/H → G · x . In particular,the derivative d φ at 1 ∈ G gives an isomorphism from g / h to the tangent space T x ( G · x ), and it iseasily checked that d φ is H -equivariant. It follows that d κ x gives an isomorphism of H -modules from to T x ( G · x ). Now let β ∈ T x ( G · x ) ∩ T x X H . Then β is fixed by H , so β = d κ x ( α ) for some α ∈ d H . As H is separable in G , α ∈ Lie( N G ( H )). Hence β ∈ T x ( N G ( H ) · x ).To finish, it is enough by Lemma 7.4 to show that generic elements of X H belong to X . As H is aprincipal stabilizer for X and X has good dimension for the G -action, G x = H and G · x is closed forgeneric x ∈ X H . Now(7.7) dim( G x ) + dim(ker( d κ x )) ≥ H )for all x ∈ X H . But equality holds in Eqn. (7.7) for x = x , so it holds for generic x ∈ X H byLemma 2.1. This shows that G · x is separable for generic x ∈ X H , so we are done. (cid:3) The following example shows that separability does not hold automatically under the hypotheses ofTheorem 7.2, not even when X has good dimension. Example . Let G = SL p ( k ), where k has characteristic p and p >
2. Let e , . . . , e p be the standardbasis vectors for the vector space V := k p and let B be the standard nondegenerate symmetric bilinearform on k p given by B ( e i , e j ) = δ ij . Now let Y be S ( V ) ∗ , the vector space of symmetric bilinear formson k p ; then G acts on Y by ( g · B )( v, w ) = B ( g − · v, g − · w ). If B ∈ Y then B is nondegenerate ifand only if the p × p matrix with i, j -entry B ( e i , e j ) has nonzero determinant, so the subvariety X ofnondegenerate forms is open and affine. Moreover, X has good dimension since the G -orbits on X allhave the same dimension.The stabilizer G B is the special orthogonal group H := SO p ( k ), and H is G -cr as char( k ) = 2 (infact, H is contained in no proper parabolic subgroups of G , so H is “ G -irreducible”). It is easily seenthat X H = { cB | c ∈ k ∗ } and N G ( H ) = H ; hence N G ( H ) acts trivially on X H . Moreover, X = G · X H .Hence H is a principal stabilizer and X has good dimension for the action of G on X .Let 0 = B = cB ∈ X H . Define λ ∈ Y ( G ) by λ ( a ) = diag( a − , . . . , a − , a p − ) (the diagonal matrixwith given entries with respect to the basis e , . . . , e p ). Let B ∈ Y be the degenerate form given by B ( a e + · · · + a p e p , b e + · · · + b p e p ) = ca p b p . Then for all a ∈ k ∗ , λ ( a ) · B = a B + ( a − p − a ) B . As X is open in Y , we may identify T B X with T B Y . Making the usual identification of the tangent spaces T k ∗ and T B Y with k and Y , respectively, we see that d κ B (1) = 2 B (note that since char( k ) = p , we have dd a ( a − p − a ) (cid:12)(cid:12)(cid:12)(cid:12) a =1 = 0). Now d κ B (1) belongs to T B ( G · B ) andto T B ( X H ), but not to T B ( N G ( H ) · B ) since the latter tangent space is zero. It follows from Lemma 7.4that ψ X,H is not separable. 8.
Examples
The constructions in Lemma 4.3 demonstrate the failure of Theorem 1.1 when the hypothesis ofcomplete reducibility is removed. In this section we provide some concrete and straightforward examplesof this phenomenon.
Example . Let the characteristic be 2 and let ρ : SL ( k ) → SL ( k ) be the adjoint representation ofSL ( k ). Concretely, let e = (cid:18) (cid:19) , h = (cid:18) (cid:19) , f = (cid:18) (cid:19) be the standard basis for X := Lie(SL ( k )) and let SL ( k ) act on X by conjugation. Then, with respectto this basis, we have ρ (cid:18) a bc d (cid:19) = a b ac bdc d . Let H be the image of ρ inside G = SL ( k ) with natural module X . Then H is reductive, but H is not G -cr since the representation ρ is not semisimple: the H -fixed subspace of X spanned by the vector h has no H -stable complement. Since H is reductive, N G ( H ) = H C G ( H ) . Direct calculation showsthat C G ( H ) is finite and hence N G ( H ) /H is finite. Now the vector h is H -fixed but has a non-closed G -orbit, since if we let λ ∈ Y ( G ) be the cocharacter defined by λ ( a ) := a
00 0 a − or each a ∈ k ∗ , then λ ( a ) · h = ah , so lim a → λ ( a ) · h = 0. It is obvious that 0 is not G -conjugate to h .Note that the same reasoning works for any nonzero multiple of h . On the other hand, the N G ( H )-orbitof any nonzero multiple of h is finite (and hence closed), and there are therefore infinitely many suchclosed N G ( H )-orbits. Hence the fibre of ψ X,H over π X,G (0) is infinite.Note that this example only works in characteristic 2 because it relies on the existence of the H -fixedvector h . This is consistent with the results above, since away from characteristic 2 the image of theadjoint representation of SL ( k ) in SL ( k ) is completely reducible—actually, it is irreducible—and henceis SL ( k )-cr. Example . We now provide an infinite family of examples generalizing the previous one. In theseexamples, G is SL m ( k ) acting on its natural module X , and H is the image of some reductive groupunder a representation in SL( X ). Since G has only one closed orbit in X (the orbit { } ), the quotient X/G is just a single point.First we consider polynomial representations of GL n ( k ) where k is an algebraically closed field ofpositive characteristic p . A good reference for the polynomial representation theory of GL n ( k ) is themonograph [21]. Further details may also be found in the monograph [20]. (To apply this here oneshould take q = 1 in the set-up considered there.)Let the characteristic be p > G = GL n ( k ) be the group of n × n -invertible matrices. Theirreducible polynomial representations of G are parametrized by partitions with at most n parts. Moreprecisely, let Λ + ( n ) be the set of partitions λ = ( λ , . . . , λ n ) with λ ≥ · · · ≥ λ n ≥
0. We may regard λ as a weight of the standard maximal torus of G : we set λ ( t ) = t λ . . . t λ n n . Then for each λ ∈ Λ + ( n )there exists an irreducible polynomial G -module L ( λ ) such that L ( λ ) has unique highest weight λ and λ occurs as a weight with multiplicity one. The modules L ( λ ) , λ ∈ Λ + ( n ), form a complete set of pairwisenon-isomorphic polynomial irreducible G -modules. We write T for the maximal torus of G consisting ofdiagonal matrices and B for the subgroup of G consisting of all invertible lower triangular matrices. Weshall also need modules induced from B to G . We denote by k λ the 1-dimensional rational T -module onwhich t ∈ T acts as multiplication by λ ( t ). The action of T on k λ extends to an action of B . For each λ ∈ Λ + ( n ) the induced module ∇ ( λ ) := ind GB k λ is a non-zero polynomial representation of G . Then ∇ ( λ ) is finite-dimensional and contains the irreducible module L ( λ ): in fact the G -socle of ∇ ( λ ) is L ( λ ).We consider the induced GL n ( k )-module ∇ ( n ( p − ∇ ( n ( p − S n ( p − E , where S n ( p − E is the n ( p − n ( k )-module E .By [19, Lem. 3.3] and [20, 4.3, (10)], the GL n ( k )-module ∇ ( n ( p − L ( p − , . . . , p − p − D = L (1 , . . . ,
1) of GL n ( k ). Now let ∆( n ( p − n ( p − ∇ ( n ( p − ∇ ( n ( p − n ( p − GL n ( k ) (∆( n ( p − L ( p − , . . . , p −
1) = D ⊗ ( p − . Now consider ∆( n ( p − n ( k )-module in the usual way. As an SL n ( k )-module, ∆( n ( p − n ( p − SL n ( k ) (∆( n ( p − L (0) = k is the trivial SL n ( k )-module.Moreover, since ∆( n ( p − n ( k )-module we have that L (0) appears as acomposition factor of ∆( n ( p − n ( k )-module ∆( n ( p − ρ : SL n ( k ) → SL m ( k ) , where m = dim(∆( n ( p − (cid:0) np − np − n (cid:1) . Let X = ∆( n ( p − H be the image of ρ inside G = SL m ( k ) = SL( X ). The previous reasoning shows that X is an indecomposable H -module andthe trivial module appears in the H -socle of X . The group H is reductive but not G -cr since therepresentation X is not semisimple.Since H is reductive we have that N G ( H ) = H C G ( H ) . Moreover, End H ( X ) = End SL n ( k ) ( X ) = k (see [26, Prop. 2.8]); this implies that C G ( H ) is finite, so N G ( H ) /H is finite.Now the quotient X H /N G ( H ) is infinite since H fixes a full 1-dimensional subspace of X and N G ( H ) /H is finite. On the other hand, the quotient X/G is a single point and so the morphism ψ X,H : X H /N G ( H ) → X/G is not a finite morphism.Note that Example 8.1 above is just this one with p = n = 2. xample . We provide another example, this time with a symplectic group. Let p = 2 and considerthe symplectic group Sp ( k ). We choose the simple roots α = (2 , −
1) and β = ( − , ( k )-module L (0 , , , , (2 , − , ( − , , (0 , − ,
1) corresponding to (0 , L (0 ,
1) and it fits into the short exact sequence0 → k → ∆(0 , → L (0 , → , where k is the trivial Sp ( k )-module.Now consider the matrix representation corresponding to the Sp ( k )-module ∆(0 , ρ : Sp ( k ) → SL ( k ) . Let X = ∆(0 ,
1) and let H be the image of Sp ( k ) in G = SL ( k ) with natural module X . Then X is an indecomposable H -module and the trivial module appears in the H -socle of X . The group H isreductive but not G -cr since the representation X is not semisimple.Since H is reductive we have that N G ( H ) = H C G ( H ) . Moreover, we have that End H ( X ) =End Sp ( k ) ( X ) = k (see [26, Prop. 2.8]), so the only endomorphisms of X as an H -module are thescalars. Since G = SL ( k ), this means that C G ( H ) is finite and so N G ( H ) /H is finite. Now, as in ourprevious examples, the quotient X H /N G ( H ) is infinite since H fixes a full one-dimensional subspaceof X and N G ( H ) /H is finite, whereas the quotient X/G is a single point. Therefore the morphism ψ X,H : X H /N G ( H ) → X/G is not a finite morphism.
Example . The above examples show that if H is the image of a non-completely reducible repre-sentation of a reductive group in G = GL( X ) or SL( X ) then the conclusion of Theorem 1.1 can fail.On the other hand, if H is the image of a completely reducible representation then we get an easyrepresentation-theoretic proof of Theorem 1.1 in this special case, as follows. If the representation istrivial (of any dimension), so that H is the trivial group, then X H = X and N G ( H ) = G , so the map ψ X,H is the identity map. If the representation is non-trivial and irreducible, then X H = { } and themap ψ X,H : X H /N G ( H ) → X/G is just the map from a singleton set to a singleton set and hence isfinite. If the representation is non-trivial and completely reducible but not irreducible then X H has an H -complement in X : say, X = X H ⊕ W . The centre of the Levi subgroup of G corresponding to thegiven decomposition normalizes H and acts as scalars on X H , so X H /N G ( H ) is again a singleton setand ψ X,H is finite. 9.
Double cosets
In this section we consider a separate but related problem, using techniques from earlier sections. Fixa reductive group G , and reductive subgroups H and K of G . The group H × K acts on G by the formula( h, k ) · g = hgk − ; the orbits of the action are the ( H, K )-double cosets and we call this action the doublecoset action . The stabilizer ( H × K ) g is given by { ( h, g − hg ) | h ∈ H ∩ gKg − } . We are interested inthe following question: when does G have good dimension for the double coset action? Note that, again,in characteristic 0 this problem was solved by Luna in [31]; he showed using ´etale slices that G alwayshas good dimension for the double coset action. The problem of translating Luna’s results to positivecharacteristic was also studied by Brundan [12, 13, 14, 16], who considered in particular the question ofwhen there is a dense double coset in G . Our main result gives a necessary and sufficient condition for G to have good dimension for the double coset action in terms of the stabilizers of the action. Theorem 9.1.
Let G be connected. The following are equivalent: (i) G has good dimension for the ( H × K ) -action; (ii) generic stabilizers of H × K on G are reductive; (iii) H ∩ gKg − is reductive for generic g ∈ G .Remarks . (i). It follows from [37, Thm. 1.1] that in order to show that generic stabilizers arereductive, it is enough to show that ( H × K ) g has minimal dimension and is reductive for some g ∈ G .(ii). Work of Popov [45] implies that if a connected semisimple group G acts on a smooth irreducibleaffine variety V and the divisor class group Cl( V ) has no elements of infinite order then generic orbitsof G on V are closed if and only if generic stabilizers of G on V are reductive. By work of Tange [59,Thm. 1.1], if G is connected then Cl( G ) has no elements of infinite order, so Theorem 9.1 follows if H and K are connected and semisimple. e need some preparatory results and notation. First, given a cocharacter τ = ( λ, µ ) ∈ Y ( H × K )and g ∈ G , we say that τ destabilizes g if lim a → τ ( a ) · g = lim a → λ ( a ) gµ ( a ) − exists. Given g ∈ G ,define a homomorphism b φ g : G → G × G by b φ g ( g ′ ) = ( g ′ , g − g ′ g ). A short calculation shows that b φ g induces an isomorphism φ g : H ∩ gKg − → ( H × K ) g . This shows that (ii) and (iii) of Theorem 9.1 areequivalent. Moreover, given g ∈ G we define an isomorphism of varieties r g : G → G by r g ( g ′ ) = g ′ g − and an isomorphism of algebraic groups ψ g : H × K → H × gKg − by ψ g ( h, k ) = ( h, gkg − ); then r g is a ψ g -equivariant map from the ( H × K )-variety G to the ( H × gKg − )-variety G , where we let H × gKg − act on G by the double coset action. Lemma 9.3.
Let g ∈ G . Then G has good dimension for the ( H × K ) -action if and only if G has gooddimension for the H × gKg − -action, and generic stabilizers of H × K on G are reductive if and only ifgeneric stabilizers of H × gKg − on G are reductive.Proof. The ψ g -equivariance of r g implies that ( H × gKg − ) · r g ( g ′ ) = r g (( H × K ) · g ′ ) and r g (( H × K ) g ′ ) =( H × gKg − ) r g ( g ′ ) for all g ′ ∈ G . The result follows. (cid:3) In the special case when A is reductive, the next result is [35, Lem. 4.1]. We take the opportunity tocorrect the proof given in loc. cit. Lemma 9.4.
Let k ′ be an algebraically closed extension field of k . Let A be a linear algebraic groupacting on an affine variety X , and let A ′ (resp. X ′ ) be the group (resp. variety) over k ′ obtained from K (resp. X ) by extension of scalars. Let x ∈ X . Then: (i) dim k ′ ( A ′ x ) = dim k ( A x ) and dim k ′ ( A ′ · x ) = dim k ( A · x ) ; (ii) A ′ · x is closed in X ′ if and only if A · x is closed in X .Proof. We regard X as a subset of X ′ and A as a subgroup of A ′ in the obvious way. The orbit map κ x : A ′ → A ′ · x is defined over k , so the closure A ′ · x (in X ′ ) is k -defined [10, Cors. AG.14.5 andAG.14.6]. This implies that A ′ · x ∩ X = A · x , where the RHS is the closure in X . The stabilizer A ′ x is k -defined—in fact, A ′ x is naturally isomorphic to the group over k ′ obtained from A x by extensionof scalars. Hence dim k ′ ( A ′ x ) = dim k ( A x ). This proves the first assertion of (i), and the second followsimmediately.Let r = dim k ′ ( A ′ · x ) = dim k ( A · x ). Set X ′ t = { y ′ ∈ X ′ | dim k ′ ( A ′ y ′ ) ≥ t } and X t = { y ∈ X | dim k ( A y ) ≥ t } for t ≥
0. Then X ′ t and X t are closed in X ′ and X , respectively, and it follows fromthe proof of [44, Lem. 3.7(c)] that X ′ t is k -defined. By (i), X t = X ′ t ∩ X . Now A ′ · x is the union of A ′ · x with certain other A ′ -orbits, each of which has dimension strictly less than r , and likewise for A · x . Hence A ′ · x (resp., A · x ) is closed if and only if A ′ · x ∩ X ′ r +1 = ∅ (resp., A · x ∩ X r +1 = ∅ ).But A ′ · x ∩ X ′ r +1 is k -defined and k is algebraically closed, so A ′ · x ∩ X ′ r +1 is empty if and only if( A ′ · x ∩ X ′ r +1 ) ∩ X = A · x ∩ X r +1 is empty. Part (ii) now follows. (cid:3) Lemma 9.5.
Assume G is connected. Let k ′ be an algebraically closed extension field of k and let G ′ , H ′ and K ′ be the algebraic groups over k ′ obtained from G , H and K , respectively, by extension of scalars.Then: (i) generic stabilizers of H ′ × K ′ on G ′ are reductive if and only if generic stabilizers of H × K arereductive; (ii) G ′ has good dimension for the ( H ′ × K ′ ) -action if and only if G has good dimension for the ( H × K ) -action.Proof. We can regard G , H and K as dense subgroups of G ′ , H ′ and K ′ , respectively. If g ∈ G then( H ′ × K ′ ) g is isomorphic to the group obtained from ( H × K ) g by extension of scalars, so ( H ′ × K ′ ) g isreductive if and only if ( H × K ) g is reductive. By Lemma 9.4, ( H ′ × K ′ ) · g is closed in G ′ if and onlyif ( H × K ) · g is closed in G , and dim(( H ′ × K ′ ) g ) is minimal if and only if dim(( H × K ) g ) is minimal,so g is a stable point for the ( H ′ × K ′ )-action if and only if it is a stable point for the ( H × K )-action.The union of the stable ( H × K )-orbits is open in G , and likewise for H ′ × K ′ and G ′ (Lemma 2.6). Theunion of the ( H × K )-orbits of minimum dimension having reductive stabilizer is open in G , and likewisefor H ′ × K ′ and G ′ [37, Thm. 1.1]. Putting these facts together, we obtain the desired result. (cid:3) Lemma 9.6.
Let λ ∈ Y ( H ) , µ ∈ Y ( K ) . Given g ∈ G such that ( λ, µ ) destabilizes g , set g :=lim a → λ ( a ) gµ ( a ) − and let u = gg − . Then µ = g − · λ and u ∈ R u ( P λ ) . roof. Since g is obtained as a limit along ( λ, µ ), we have that ( λ, µ ) fixes g , so λ ( a ) g µ ( a ) − = g forall a ∈ k ∗ . Rearranging, we see that µ = g − · λ . Now for all a ∈ k ∗ , λ ( a ) gµ ( a ) − = λ ( a ) ug µ ( a ) − = λ ( a ) ug ( g − λ ( a ) − g ) = λ ( a ) uλ ( a ) − g . As lim a → λ ( a ) gµ ( a ) − = g , it follows that lim a → λ ( a ) uλ ( a ) − = 1, so u ∈ R u ( P λ ). (cid:3) Lemma 9.7.
Assume G is connected. Let G = G/Z ( G ) , let σ : G → G be the canonical projectionand set H = σ ( H ) and K = σ ( K ) . Then for all g ∈ G : (i) ( H × K ) g is reductive if and only if ( H × K ) σ ( g ) is reductive; (ii) if ( H × K ) · σ ( g ) is closed then ( H × K ) · g is closed.Proof. (i). Let e H = HZ ( G ) and let e K = KZ ( G ) . Let A = ( σ × σ ) − (( H × K ) σ ( g ) ), a subgroupof e H × e K . Define ψ : A → G by ψ ( e h, e k ) = g − e hg e k − . A short calculation shows that ψ gives ahomomorphism from A to Z ( G ) , with kernel ( H × K ) g . Moreover, σ × σ gives an epimorphism from A to ( H × K ) σ ( g ) , with kernel Z ( G ) × Z ( G ) . Part (i) now follows.(ii). Let g ∈ G and suppose ( H × K ) · σ ( g ) is closed. Let ( λ, µ ) ∈ Y ( H × K ) such that g ′ :=lim a → λ ( a ) gµ ( a ) − exists. Set g = σ ( g ), g ′ = σ ( g ′ ). Let λ = σ ◦ λ ∈ Y ( H ) and µ = σ ◦ µ ∈ Y ( K );then g ′ = lim a → λ ( a ) g µ ( a ) − . By hypothesis, g ′ is ( H × K )-conjugate to g . Now the group Z ( G ) acts on G by right inverse multiplication, and we can identify σ with the canonical projection to thequotient. The orbits of Z ( G ) all have the same dimension, so σ is a geometric quotient. Moreover, the Z ( G ) -action commutes with the ( H × K )-action, so H × K acts on G . By construction, ( h, k ) · σ ( x ) =( σ ( h ) , σ ( k )) · σ ( x ) for all x ∈ G , h ∈ H and k ∈ K . In particular, g ′ is ( H × K )-conjugate to g . Itfollows from [9, Cor. 3.5(ii)] that g ′ is ( H × K )-conjugate to g . Hence ( H × K ) · g is closed. This proves(ii). (cid:3) Lemma 9.8.
Suppose G , H and K are connected. Let λ ∈ Y ( H ) . Suppose there exists a nonemptysubset C of G such that ( H × K ) · C is open and has the following property: for all g ∈ C , there exists τ g = ( λ, µ g ) ∈ Y ( H × K ) such that τ g destabilizes g . (i) There exists g ∈ G such that λ ∈ Y ( g Kg − ) and ( H × g Kg − ) · P λ is dense in G . Moreover,for all g ∈ P λ , the cocharacter ( λ, λ ) of H × g Kg − destabilizes g . (ii) Suppose in addition that τ g fixes g for all g ∈ C . Then ( H × g Kg − ) · L λ is dense in G , and ( λ, λ ) fixes every l ∈ L λ .Proof. Fix v ∈ C and let v := lim a → τ v ( a ) · v = lim a → λ ( a ) vµ v ( a ). Then λ = v · µ v by Lemma 9.6,so λ ∈ Y ( v Kv − ). The equivariance of r v implies that for any w ∈ C , ( λ, v · µ w ) ∈ Y ( H × v Kv − )destabilizes wv − to w v − , where w := lim a → τ w ( a ) · w . By Lemma 9.3, we can replace K with v Kv − and C with Cv − . So without loss we assume that λ ∈ Y ( K ).Let g ∈ C . By hypothesis, τ g = ( λ, µ g ) destabilizes g . As im( λ ) is contained in K , there exists k ∈ K such that µ := k · µ g commutes with λ . Set g = gk − = (1 , k ) · g , so that ( λ, µ ) destabilizes g . Finally,set g = lim a → λ ( a ) g µ ( a ) − . Then λ = g · µ by Lemma 9.6. Fix a maximal torus T of G such that λ, µ ∈ Y ( T ) and let n , . . . , n r ∈ N G ( T ) be a set of representatives for the Weyl group N G ( T ) /T . Now g T g − is a maximal torus of L λ , so by conjugacy of maximal tori in L λ , we have xg T g − x − = T for some x ∈ L λ . Then xg = tn i for some i and some t ∈ T , so g = ln i , where l := x − t ∈ L λ . ByLemma 9.6, we have g = ug = uln i for some u ∈ R u ( P λ ), so g ∈ P λ n i and g = g k ∈ ( H × K ) · ( P λ n i ).Since g ∈ C was arbitrary, it now follows that S ri =1 ( H × K ) · ( P λ n i ) contains ( H × K ) · C and, since G is connected, ( H × K ) · ( P λ n i ) is dense in G for at least one i . Note also that λ = g · µ = ln i · µ , so µ = n − i l − · λ = n − i · λ , so λ = n i · µ ∈ Y ( n i Kn − i ).Keeping the notation in the previous paragraph, for each i , ( H × K ) · ( P λ n i ) is constructible, so( H × K ) · ( P λ n i ) is either dense or contained in a proper closed subset of G . Thus the union of thosesubsets ( H × K ) · ( P λ n i ) that are dense contains an open subset of G ; note also that this union is( H × K )-stable. Since ( H × K ) · C is open, we can find g ′ ∈ C such that for any i , if g ′ ∈ ( H × K ) · ( P λ n i )then ( H × K ) · ( P λ n i ) is dense. By the arguments in the paragraph above applied to g ′ , there exists i such that g ′ ∈ ( H × K ) · ( P λ n i ) and for this i we have λ = n i · µ ∈ Y ( n i Kn − i ); moreover, ( H × K ) · ( P λ n i )is dense by construction. It follows that ( H × n i Kn − i ) · P λ = r n i (( H × K ) · ( P λ n i )) is dense in G , sothe first assertion of part (i) follows with g = n i . It is obvious that ( λ, λ ) destabilizes g for all g ∈ P λ ,so we have proved part (i).If g ∈ C and τ g fixes g then ( λ, µ ) fixes g , so g = g ∈ L λ n i for some i . The first assertion of(ii) follows by a similar argument to that above but applied to S ri =1 ( H × K ) · ( L λ n i ), and the secondassertion is again obvious. (cid:3) roof of Theorem 9.1. We have shown already that (ii) and (iii) are equivalent, so it is enough to provethat (i) and (ii) are equivalent. First note that for any g ∈ G , ( H ∩ K ) · g is closed if and only if( H ∩ K ) · g = ( H ∩ K ) · g is closed, and H ∩ gKg − is reductive if and only if ( H ∩ gKg − ) =( H ∩ gK g − ) is reductive, which is the case if and only if H ∩ gK g − is reductive. Hence we canassume that H and K are connected. Moreover, we can assume by Lemma 9.5 that k is uncountable.The implication (i) = ⇒ (ii) follows immediately from Lemma 3.3(ii). For the reverse implication,we use induction on dim( G ). Suppose generic stabilizers are reductive. The result is immediate ifdim( G ) = 0. If G is not semisimple then let G , σ , H and K be as in Lemma 9.7. Then genericstabilizers of H × K on G are reductive, by Lemma 9.7(i). Since dim( G ) < dim( G ), it follows byinduction that generic orbits of H × K on G are closed. Part (ii) of Lemma 9.7 now implies thatgeneric orbits of H × K on G are closed, so we are done. Hence we can assume that G is semisimple.First we consider the case when generic stabilizers of H × K on G are positive-dimensional. Thenall stabilizers of H × K on G are positive-dimensional, by semi-continuity of stabilizer dimension. Foreach g ∈ G such that ( H × K ) g is reductive, choose a nontrivial cocharacter τ g ∈ Y (( H × K ) g ). Thefixed point set G τ g := G im( τ g ) is closed, so C g := ( H × K ) · G τ g is constructible. Since generic stabilizersof H × K on G are reductive, the constructible sets C g for g ∈ G such that ( H × K ) g is reductivecover an open dense subset U of G , by [37, Thm. 1.1]. There are only countably many of these sets,as H × K has only countably many conjugacy classes of cocharacters. By [37, Cor. 2.5], C e g is dense in G for some e g ∈ G . Hence there exists τ = ( λ, µ ) ∈ Y ( H × K ) such that for generic g ∈ G , g is fixedby an ( H × K )-conjugate of τ . It follows from Lemma 9.8 that for some g ∈ G , λ ∈ Y ( g Kg − ) and( H × g Kg − ) · L λ is dense in G . By Lemma 9.3, there is no harm in assuming that g Kg − = K —i.e.,that λ ∈ Y ( K ) and ( H × K ) · L λ is dense in G —and we shall do this for notational convenience.To prove that generic ( H × K )-orbits on G are closed, it is therefore enough to show that ( H × K ) · l isclosed for generic l ∈ L λ . Let H = L λ ( H ) and let K = L λ ( K ); then H × K = L ( λ,λ ) ( H × K ). Considerthe double coset action of H × K on L λ . Let l ∈ L λ . Then ( λ, λ ) fixes l , so ( H × K ) l = L ( λ,λ ) (( H × K ) l ),which is reductive if ( H × K ) l is. Hence generic stabilizers of ( H × K ) on L λ are reductive. As G issemisimple, dim( L λ ) < dim( G ), so generic ( H × K )-orbits on L λ are closed by induction. It followsfrom Remark 4.2(iii) that ( H × K ) · l is closed for generic l ∈ L λ , so we are done as ( H × K ) · L λ is densein G .Now consider the case when generic stabilizers of H × K on G are finite. Suppose generic ( H × K )-orbits on G are not closed. Then G cl is a proper closed subset of G , so the union of the non-closed orbitscontains a nonempty open subset of G . For each g ∈ G such that ( H × K ) · g is not closed, choose nontrivial τ g ∈ Y ( H × K ) such that τ g destabilizes g . By an argument similar to the one in the positive-dimensionalcase above, there exist λ ∈ Y ( H ) and g ∈ G such that λ ∈ Y ( g Kg − ) and ( H × g Kg − ) · P λ is densein G . As before, we can assume that g Kg − = K . Now R u ( P − λ )( H ) P λ ( H ) and P λ ( K ) R u ( P − λ )( K ) arenonempty open subsets of H and K respectively [10, Prop. 14.21(iii)], so R u ( P − λ )( H ) P λ R u ( P − λ )( K ) isdense in G , as HP λ K is. It follows that dim( R u ( P − λ )( H )) + dim( R u ( P − λ )( K )) + dim( P λ ) ≥ dim( G ),so dim( R u ( P − λ )( H )) + dim( R u ( P − λ )( K )) ≥ dim( G ) − dim( P λ ) = dim( R u ( P λ )).By hypothesis, we can choose g ∈ P λ such that ( H × K ) g is finite. Write g = ul , where l = L λ and u ∈ R u ( P λ ); then l = lim a → ( λ, λ )( a ) · g . We show that l is ( H × K )-conjugate to g . Considerthe double coset action of R u ( P λ ( H )) × R u ( P λ ( K )) on G . Let O = ( R u ( P λ ( H )) × R u ( P λ ( K ))) · g andconsider the orbit map κ g : R u ( P λ ( H )) × R u ( P λ ( K )) → O given by κ g ( h, k ) = hgk − . It is clear that O ⊆ R u ( P λ ) l . Note that O is closed, since O is the orbit of an action of a unipotent group on an affinevariety [10, Prop. 4.10]. The stabilizer of g in R u ( P λ ( H )) × R u ( P λ ( K )) is finite, since ( H × K ) g is finite,so O has dimension dim( R u ( P λ ( H ))) + dim( R u ( P λ ( K ))). Now dim( R u ( P λ ( H ))) + dim( R u ( P λ ( K ))) =dim( R u ( P − λ )( H )) + dim( R u ( P − λ )( K )) ≥ dim( R u ( P λ )), and since O is closed, this forces O to be thewhole of R u ( P λ ) l . Hence there exists ( h, k ) ∈ R u ( P λ ( H )) × R u ( P λ ( K )) such that ( h, k ) · g = l , asrequired.Now ( H × K ) l is finite, since l is ( H × K )-conjugate to g . But ( λ, λ ) fixes l , a contradiction. Wededuce that generic ( H × K )-orbits on G are closed after all. This completes the proof. (cid:3) Remark . One can prove the following more general statement of Theorem 9.1 for non-connectedreductive G . Let G , . . . , G r be the minimal subsets of G having the property that each G i is ( H × K )-stable and contains some connected component of G . Each G i is a union of certain connected componentsof G ; if H and K are connected then the G i are precisely the connected components of G . Here is ourresult: for each i , G i has good dimension for the ( H × K )-action if and only if generic stabilizers of H × K on G i are reductive if and only if H ∩ gKg − is reductive for generic g ∈ G i . To see this, note first that e can assume that H and K are connected, by the proof of Theorem 9.1; hence we can assume thateach G i is a connected component of G . We can now choose g ∈ G such that G i g = G , and use themap r g to translate the case of G i into the case of the connected group G (cf. the proof of Lemma 9.3).We leave the details to the reader.We record a useful corollary. Corollary 9.10.
Suppose one of H and K is a torus. Then G has good dimension for the ( H × K ) -action.Proof. This is immediate from Theorem 9.1, since any subgroup of a torus is reductive. (cid:3)
We now consider a concrete example; our methods allow us to deal with arbitrary characteristic. Notethat we use Theorem 1.1 in parts (a) and (b) below.
Example . Let G be simple of type B and fix a maximal torus T of G . Let A be the subgroup of G generated by the long root groups with respect to T . If p = 2 then let B be the subgroup of G generatedby the short root groups with respect to T . The groups A and B are normalized by N G ( T ).(a). Let p be arbitrary and let H = K = A . Since dim( G ) = 10 and dim( H ) = dim( K ) = 6,dim( H × K ) g ≥ g ∈ G , with equality if and only if ( H × K ) · g is dense in G . Let λ ∈ Y ( T ) benontrivial. We show first that for generic l ∈ L λ , ( H × K ) · l is closed. If L λ = T or L λ is a long-rootLevi subgroup (that is, a Levi subgroup L such that [ L, L ] is the subgroup of type A corresponding tosome long root) then L λ ≤ A , so ( H × K ) · l = A is closed. Note that in this case, ( H × K ) · L λ = A isnot dense in G .So suppose L λ is a short-root Levi subgroup. As in the positive-dimensional case in the proof ofTheorem 9.1, it is enough to show that ( L λ ( H ) × L λ ( K )) · l is closed for generic l ∈ L λ . But this followsfrom Corollary 9.10, since L λ ( H ) × L λ ( K ) = T × T is a torus. Moreover, in this case the quotient space L λ / ( T × T ) is positive-dimensional, as dim( T × T ) = 4 = dim( L λ ) and ( T × T ) l has dimension at least 1for all l ∈ L (since ( λ, λ ) fixes l ). It follows that the quotient space G/ ( H × K ) is positive-dimensional.To see this, let S be the image of ( λ, λ ); note that T × T ≤ N H × K ( S ) ≤ N H × K ( T × T ). Now considerthe maps L λ / ( T × T ) → L λ /N H × K ( S ) → G/ ( H × K ). The first map is finite as T × T has finiteindex in N H × K ( S ), while the second is finite by Theorem 1.1 (applied to the subgroup S of H × K ), sodim( G/ ( H × K )) ≥ dim( L λ / ( T × T )) ≥
1, as claimed.Next we show that for generic g ∈ G , ( H × K ) g contains a nontrivial torus. Suppose not. Thenfor generic g ∈ G , ( H × K ) g is a unipotent subgroup of H of dimension at least 2, so ( H × K ) g is amaximal unipotent subgroup of H and has dimension 2. But then the orbit ( H × K ) · g is dense in G ,so G/ ( H × K ) is a single point, which is a contradiction.It follows from the proof of the positive-dimensional case of Theorem 9.1 that ( H × n i Kn − ) · L λ isdense in G for some nontrivial λ ∈ Y ( T ) and some i . But N G ( T ) normalizes K , so ( H × K ) · L λ isdense in G (and hence L λ is a short-root Levi subgroup of G ). We deduce from the discussion abovethat generic ( H × K )-orbits in G are closed. Moreover, we see that the map L λ / ( T × T ) → G/ ( H × K )is finite and dominant, hence surjective. A simple calculation shows that generic stabilizers of T × T on L λ have dimension 1, so dim( L λ / ( T × T )) = 1, which implies that dim( G/ ( H × K )) = 1. Hencegeneric stabilizers of H × K on G are reductive groups of dimension 3. It follows that for generic g ∈ G ,( H × K ) g is of type A .(b). Let p = 2 and let H = K = B . Then generic orbits of H × K on G are closed and for generic g ∈ G , ( H × M ) g is of type A . The proof is similar to case (a).(c). Let p = 2, let H = A and let K = B . Consider the stabilizer ( H × K ) , which is isomorphic viathe map φ to H ∩ K . It is easily seen that H ∩ K = T , so ( H × K ) = { ( t, t ) | t ∈ T } . It follows thatdim(( H × K ) ·
1) = dim( H × K ) − dim( T ) = 12 − G ), so ( H × K ) · G andgeneric stabilizers have dimension 2 and are reductive. It follows from Theorem 9.1 that generic orbitsare closed. Hence ( H × K ) · H × K acts transitively on G . (This conclusion also followsfrom [14, Thm. A], since A and B are maximal connected subgroups of G .)We finish the section with a further example. Example . Suppose p = 2 and let G be simple and of rank r . Let τ ∈ Aut( G ) be an involution thatinverts a maximal torus of G —such a τ always exists, by [17, Lem. 3.6]—and let H = K = C G ( τ ). Then( H × K ) g = H ∩ gKg − is a finite group of order 2 r for generic g ∈ G [17, Thm. 9], so G has gooddimension for the H × K -action, by Theorem 9.1. (Note that in [17, Sec. 3.2] one considers the action of H by left multiplication on G/K rather than the double coset action of H × K on G , but the argumentscarry over easily to our setting. See also [37, Ex. 8.4].) e now consider a striking feature of this example: namely that, although generic stabilizers for thedouble coset action are nontrivial, there is a unique ( H × K )-orbit O consisting of elements with trivialstabilizer [17, Thm. 9] (compare Example 3.2). If p = 0 then O cannot be closed: for otherwise therewould exist an ´etale slice through any element of O , so every element in some open neighbourhood of O would have trivial stabilizer by the argument of Example 3.2, a contradiction. More generally, forarbitrary p = 2 the argument of [37, Ex. 8.4] implies that G has a principal stabilizer A which is a finitegroup of order 2 r , and it follows from Theorem 1.1 that if g ′ ∈ G and ( H × K ) · g ′ is closed then ( H × K ) g ′ contains a conjugate of A . Hence we see again that O cannot be closed.We give a direct proof of this. The orbit O is of the form ( H × K ) · g , where g ∈ G has the propertythat u := τ τ g is a regular unipotent element of G and u is inverted by τ (see [17, Prop. 3.1]). In fact, wecan choose g to be a regular unipotent element of G such that g = u and τ inverts g (take g to be u s if p >
0, where 2 s ≡ | u | ). Set U = h g i ; then τ normalizes U , as τ inverts g . There exists λ ∈ Y ( G )such that lim a → λ ( a ) g ′ λ ( a ) − = 1 for all g ′ ∈ U . We can choose λ to be optimal in the sense of [9, Defn.4.4 and Thm. 4.5] (cf. Section 2.5). Then τ normalises P λ . Now N Aut( G ) ( P λ ) is an R-parabolic subgroupof the reductive group Aut( G ) [35, Prop. 5.4(a)], so N G ( P λ ) = P µ for some µ ∈ Y ( G ). As τ ∈ P µ and h τ i is linearly reductive, we can choose µ to centralize τ : that is, we can choose µ to belong to Y ( H ).Let σ = ( µ, µ ) ∈ Y ( H × K ). Then lim a → σ ( a ) · g = lim a → µ ( a ) gµ ( a ) − = 1 since U ≤ R u ( P λ ) = R u ( P µ ). But clearly 1 O , so O is not closed, as claimed.10. Applications to G -complete reducibility We finish with some applications of ideas from Sections 3 and 9 to G -complete reducibility. Our nextlemma gives a basic structural result about G and its subgroups which can quickly be proven using theframework we have now set up; the setting is as in Section 9 but more general, since we allow one of thesubgroups to be non-reductive (cf. [15]). The argument used is taken from the proof of [29, Kap. III.2.5,Satz 2]; note that although the reference [29] works with groups and varieties defined over the complexnumbers, many of the arguments are completely general. For convenience, we reproduce the details here. Lemma 10.1.
Suppose K is a subgroup of G and let H be a reductive subgroup of G that contains amaximal torus of K . Then HK is a closed subset of G .Proof. First suppose that K is unipotent. The quotient X = G/H is affine and H is the stabilizer in G of the point x = π G,H (1) ∈ X . Since K is unipotent, and all orbits for unipotent groups on affinevarieties are closed [10, Prop. 4.10], K · x is closed, so KH (and hence HK ) is closed in G by Lemma 3.5.Now, in the general case, let T be a maximal torus of K contained in H and let B be a Borel subgroupof K containing T with unipotent radical U . Then U H = BH is closed in G by the first paragraph, andthe following argument from [29, Kap. III.2.5, Satz 2] gives us what we want. We have a sequence ofmorphisms K × G φ −→ K × G ρ = π K,B × id −−−−−−−→ K/B × G pr −→ G where φ ( g ′ , g ) := ( g ′ , g ′ g ) for g ′ ∈ K , g ∈ G , π K,B is the quotient morphism K → K/B and pr is theprojection of K/B × G onto the second factor. Let Y = K × BH . Since BH is closed in G , Y is closedin K × G . Since φ is an isomorphism of varieties, φ ( Y ) is closed in K × G and therefore ρ ( φ ( Y )) is closedin K/B × G . Finally, since K/B is complete, pr ( ρ ( φ ( Y ))) is closed in G . But it is easy to see thatpr ( ρ ( φ ( Y ))) = KH , so we are done. (cid:3) The lemma above allows a quick proof of the following result.
Proposition 10.2.
Suppose G is reductive, X is an affine G -variety and x ∈ X . If H is a reductivesubgroup of G containing a maximal torus of G x , then H · x is closed in G · x . In particular, if G · x isclosed in X then H · x is closed in X .Proof. By Lemma 10.1, under the given hypotheses, HG x is closed in G . Hence, by Lemma 3.5, H · x isclosed in G · x . (cid:3) Remark . The argument of Lemma 10.1 is used in [29, Kap. III, 2.5, Folgerung 3] to show that if X is affine and G x contains a maximal torus of G , then G · x is closed, a result which has obvious similaritiesto Proposition 10.2. Corollary 10.4.
Suppose H and K are reductive subgroups of the reductive group G . If H ∩ K containsa maximal torus of H or K , then HK is closed in G and H ∩ K is a reductive group. roof. Without loss, suppose H ∩ K contains a maximal torus of K . The first conclusion is a special caseof Lemma 10.1. For the second, apply Proposition 10.2 to the action of G and H on the quotient G/K .Since
G/K is affine and H · π G,K (1) is closed in
G/K , this orbit is also affine and hence the stabilizer H π G,K (1) = H ∩ K is reductive by Lemma 3.3(ii). (cid:3) A theme running through [5] and subsequent papers on complete reducibility by the same authorsis the following general question: if A and H are subgroups of G with A ⊆ H and H reductive, whatconditions ensure that if A is G -cr then A is H -cr, and vice versa? Because of the link between completereducibility and closed orbits in G n explained in Section 2.4 above, this is readily seen to be a specialcase of the general questions considered in this paper. Since this was one of the original motivations forthe work presented here, we briefly record some of the translations of our main results into the languageof complete reducibility and give a couple of other consequences in this setting.First note that Proposition 10.2 specializes to [5, Prop. 3.19] in the setting of complete reducibility:that is, with notation as just set up, if H also contains a maximal torus of C G ( A ) and A is G -cr, then A is H -cr. More generally, we have: Proposition 10.5.
Suppose H is a reductive subgroup of G , and let A be a subgroup of H . (i) If A is H -cr, then HC G ( A ) is closed in G . (ii) If A is G -cr, then A is H -cr if and only if HC G ( A ) is closed in G .Proof. (i). Let a ∈ H n be a generic tuple for A . Suppose A is H -cr; then H · a is closed in H n . Since H n is closed in G n , H · a is closed in G · a . Therefore, by Lemma 3.5(i), HG a = HC G ( A ) is closed in G .(ii). Using a generic tuple for A again, this becomes a direct application of Lemma 3.5(ii). (cid:3) The notions of reductive pairs from [46, §
3] and separability from [5, Def. 3.27] have proved usefulin the study of complete reducibility: see [5, § Proposition 10.6.
Suppose ( G, H ) is a reductive pair. Let A be a separable subgroup of G contained in H . Then HC G ( A ) is closed in G .Proof. Let a ∈ H n be a generic tuple for A . Then C G ( A ) = G a , and since A is separable in G , the orbit G · a is separable. Now Richardson’s “tangent space argument” [46, §
3] (generalized to n -tuples in [53])shows that G · a ∩ H n decomposes into finitely many H -orbits, each of which is closed in G · a ∩ H n .Since one of these orbits is H · a , we can conclude that H · a is closed in G · a ∩ H n , and hence in G · a .Therefore, HG a = HC G ( A ) is closed in G by Lemma 3.5(i). (cid:3) Remark . Note that every pair (
G, H ) of reductive groups with H ≤ G is a reductive pair incharacteristic 0 and the separability hypothesis is also automatic. In characteristic p >
0, every subgroupof G is separable as long as p is “very good” for G ; see [7, Thm. 1.2].As a final remark, we note that there are Lie algebra analogues of Propositions 10.5 and 10.6, wherewe replace the subgroup A with a Lie subalgebra of Lie( H ). For details of how to make such translations,see [9, § References [1] P. Bardsley, R.W. Richardson, ´Etale slices for algebraic transformation groups in characteristic p , Proc. Lond. Math.Soc. (3) , no. 2 (1985), 295–317.[2] M. Bate, Optimal subgroups and applications to nilpotent elements , Transform. Groups , no. 1 (2009), 29–40.[3] M. Bate, S. Herpel, B. Martin, G. R¨ohrle, G -complete reducibility and semisimple modules , Bull. Lond. Math. Soc. , no. 6 (2011), 1069–1078.[4] , Cocharacter-closure and the rational Hilbert-Mumford Theorem , Math. Zeit. (2017), no. 1–2, 39–72.[5] M. Bate, B. Martin, G. R¨ohrle,
A geometric approach to complete reducibility , Invent. Math. , no. 1 (2005),177–218.[6] ,
Complete reducibility and commuting subgroups , J. Reine Angew. Math. (2008), 213–235.[7] M. Bate, B. Martin, G. R¨ohrle, R. Tange,
Complete reducibility and separability , Trans. Amer. Math. Soc. , no. 8(2010), 4283–4311.[8] ,
Complete reducibility and conjugacy classes of tuples in algebraic groups and Lie algebras , Math. Z. , no.3-4 (2011), 809–832.[9] ,
Closed orbits and uniform S -instability in geometric invariant theory , Trans. Amer. Math. Soc. , no. 7(2013), 3643–3673.[10] A. Borel, Linear algebraic groups , Graduate Texts in Mathematics , Springer-Verlag 1991.[11] A. Borel, J. Tits,
Groupes r´eductifs , Inst. Hautes ´Etudes Sci. Publ. Math. (1965), 55–150.[12] J. Brundan, Double coset density in reductive algebraic groups , J. Algebra (1995), no. 3, 755–767.
13] ,
Double coset density in exceptional algebraic groups , J. London Math. Soc. (2) (1998), no. 1, 63–83.[14] , Double coset density in classical algebraic groups , Trans. Amer. Math. Soc. (2000), no. 3, 1405–1436.[15] ,
Dense orbits and double cosets , Algebraic groups and their representations (Cambridge, 1997), 259–274,NATO Adv. Sci. Inst. Ser. C Math. Phys. Sci. (1998), Kluwer Acad. Publ., Dordrecht.[16]
Double cosets in algebraic groups , PhD thesis, Imperial College London, 1996.[17] T.C. Burness, R.M. Guralnick, J. Saxl,
On base sizes for algebraic groups , J. Eur. Math. Soc. (2017), no. 8,2269–2341.[18] I. Dolgachev Weighted projective varieties , Lect. Notes in Math. (1982), 34–71.[19] M. De Visscher, S. Donkin
On projective and injective polynomial modules , Math. Z. (2005), 333–358.[20] S. Donkin,
The q -Schur algebra , LMS Lecture Notes , Cambridge University Press 1998.[21] K. Erdmann, J. A. Green and M. Schocker, Polynomial representations of GL n , Second Edition with an Appendix onSchensted Correspondence and Littelmann Paths, Lecture Notes in Mathematics , Springer 2007.[22] W.J. Haboush, Reductive groups are geometrically reductive , Ann. of Math. (2) , no. 1 (1975), 67–83.[23] ,
Homogeneous vector bundles and reductive subgroups of reductive algebraic groups , Amer. J. Math. , no.6 (1978), 1123–1137.[24] W.H. Hesselink,
Uniform instability in reductive groups , J. Reine Angew. Math. (1978), 74–96.[25] J.E. Humphreys,
Linear Algebraic Groups , Springer-Verlag, New York, 1975.[26] J.C. Jantzen,
Representations of Algebraic Groups , second ed., Math. Surveys Monogr., vol 107, Amer. Math. Soc.2003.[27] ,
Nilpotent orbits in representation theory , Lie theory, 1–211, Progr. Math. (2004), Birkh¨auser Boston.[28] G.R. Kempf,
Instability in invariant theory , Ann. Math. (1978), 299–316.[29] H. Kraft,
Geometrische Methoden in der Invariantentheorie , Aspects of Mathematics, D1 , Friedr. Vieweg & Sohn,Braunschweig, 1984.[30] M.W. Liebeck, G.M. Seitz, Reductive subgroups of exceptional algebraic groups , Mem. Amer. Math. Soc. no. (1996).[31] D. Luna,
Sur les orbites ferm´ees des groupes alg´ebriques r´eductifs , Invent. Math. (1972), 1–5.[32] , Slices ´etales , Bull. Soc. Math. France, Memoire (1973), 81–105.[33] , Adh´erences d’orbite et invariants , Invent. Math. , no. 3 (1975), 231–238.[34] D. Luna, R.W. Richardson, A generalization of the Chevalley restriction theorem , Duke Math. J. , no. 3 (1979),487–496.[35] B.M.S. Martin, Reductive subgroups of reductive groups in nonzero characteristic , J. Algebra , no. 2 (2003),265–286.[36] ,
A normal subgroup of a strongly reductive subgroup is strongly reductive , J. Algebra , no. 2 (2003),669–674.[37] ,
Generic stabilisers for actions of reductive groups , Pacific J. Math. , no. 1–2 (2015), 397–422.[38] B.M.S. Martin, A. Neeman,
The map V → V//G need not be separable , Math. Res. Lett. (2001), no. 5–6, 813–817.[39] G. McNinch, Linearity for actions on vector groups , J. Algebra (2014), 666–688.[40] D. Mumford,
The red book of varieties and schemes . Lecture Notes in Math. 1358, Springer, Berlin, 1988.[41] D. Mumford, J. Fogarty, F. Kirwan,
Geometric invariant theory . Third edition. Ergebnisse der Mathematik und ihrerGrenzgebiete, 34. Springer-Verlag, Berlin, 1994.[42] M. Nagata,
Complete reducibility of rational representations of a matric group , J. Math. Kyoto Univ. (1961), 87–99.[43] , Invariants of a group in an affine ring , J. Math. Kyoto Univ. (1963/1964), 369–377.[44] P. E. Newstead, Introduction to moduli problems and orbit spaces , Tata Institute of Fundamental Research Lectureson Mathematics and Physics . Tata Institute of Fundamental Research, Bombay, 1978.[45] V.L. Popov, On the stability of the action of an algebraic group on an algebraic variety , 1972 Math. USSR Izv. ,367–379.[46] R.W. Richardson, Conjugacy classes in Lie algebras and algebraic groups , Ann. Math. (1967), 1–15.[47] , Affine coset spaces of reductive algebraic groups , Bull. London Math. Soc. , no. 1 (1977), 38–41.[48] , On orbits of algebraic groups and Lie groups , Bull. Austral. Math. Soc. , no. 1 (1982), 1–28.[49] , Conjugacy classes of n -tuples in Lie algebras and algebraic groups , Duke Math. J. , no. 1 (1988), 1–35.[50] , Principal orbit types for algebraic transformation spaces in characteristic zero , Invent. Math. (1974), 6–14.[51] G. Rousseau, Immeubles sph´eriques et th´eorie des invariants , C.R.A.S. (1978), 247–250.[52] J-P. Serre,
Compl`ete r´eductibilit´e , S´eminaire Bourbaki, 56`eme ann´ee, 2003-2004, n o Two notes on a finiteness problem in the representation theory of finite groups , Austral. Math. Soc. Lect.Ser. , Algebraic groups and Lie groups, 331–348, Cambridge Univ. Press, Cambridge, 1997.[54] T.A. Springer, Linear algebraic groups , Second edition. Progress in Mathematics, 9. Birkh¨auser Boston, Inc., Boston,MA, 1998.[55] T.A. Springer, R. Steinberg,
Conjugacy classes , Seminar on algebraic groups and related finite groups, Lecture Notesin Mathematics , Springer-Verlag, Heidelberg (1970), 167–266.[56] D.I. Stewart,
The reductive subgroups of G , J. Group Theory (2010), no. 1, 117–130.[57] , The reductive subgroups of F , Mem. Amer. Math. Soc. (2013), no. 1049, vi+88pp.[58] , On unipotent algebraic G -groups and 1-cohomology , Trans. Amer. Math. Soc. 365 (2013), no. 12, 6343–6365.[59] R. Tange, Infinitesimal invariants in a function algebra , Canad. J. Math. epartment of Mathematics, University of York, York YO10 5DD, United Kingdom E-mail address : [email protected] Department of Mathematics, University of York, York YO10 5DD, United Kingdom
E-mail address : [email protected] Department of Mathematics, University of Aberdeen, King’s College, Fraser Noble Building, AberdeenAB24 3UE, United Kingdom
E-mail address : [email protected]@abdn.ac.uk