aa r X i v : . [ m a t h . C O ] J un VEXILLARY SIGNED PERMUTATIONS REVISITED
DAVID ANDERSON AND WILLIAM FULTON
Abstract.
We study the combinatorial properties of vexillarysigned permutations, which are signed analogues of the vexillarypermutations first considered by Lascoux and Sch¨utzenberger. Wegive several equivalent characterizations of vexillary signed permu-tations, including descriptions in terms of essential sets and patternavoidance, and we relate them to the vexillary elements introducedby Billey and Lam.
Introduction
The class of vexillary permutations in S n , first identified by Lascouxand Sch¨utzenberger [LS1, LS2], plays a central role in the combinatoricsof the symmetric group and the corresponding geometry of Schubertvarieties and degeneracy loci. The name derives from the fact that theSchubert polynomial of a vexillary permutation is equal to a flaggedSchur polynomial. This was given a geometric explanation in [Fu]:vexillary permutations correspond to degeneracy loci defined by sim-ple rank conditions, whose classes are computed by variations of theKempf-Laksov determinantal formula.Degeneracy loci of other classical types are indexed by the group W n of signed permutations. In the course of proving analogous Pfaf-fian formulas for such loci [AF0, AF1], we constructed vexillary signedpermutations , starting with the notion of a triple . A triple is three s -tuples of positive integers, τ = ( k , p , q ), with k = (0 < k < · · · < k s ), p = ( p ≥ · · · ≥ p s > q = ( q ≥ · · · ≥ q s > k i +1 − k i ≤ p i − p i +1 + q i − q i +1 for 1 ≤ i ≤ s −
1. Given such a triple,one constructs a signed permutation w = w ( τ ) (see § S n have many equivalent characterizations,some of which will be reviewed below; others may be found in [Mac].The quickest one is via pattern avoidance : a permutation v is vexillary Date : June 1, 2016.DA was partially supported by NSF Grant DMS-1502201 and a postdoctoralfellowship from the Instituto Nacional de Matem´atica Pura e Aplicada (IMPA). if and only if it avoids the pattern [2 1 4 3]—that is, there are noindices a < b < c < d such that v ( b ) < v ( a ) < v ( d ) < v ( c ). Anotheris that the Stanley symmetric function of a vexillary permutation isequal to a single Schur function. The latter property was taken as thestarting point for Billey and Lam’s extension of “vexillary” to other Lietypes: they defined three distinct classes of vexillary elements in typesB, C, and D, whose Stanley functions are equal to single Schur P - or Q -functions [BL]. Our starting point is the geometric property: thevexillary signed permutations considered in [AF0, AF1] correspond todegeneracy loci defined by rank conditions of a particularly simple kind,and whose Schubert polynomials can be written as flagged Pfaffians.As often happens, properties that coincide in type A diverge in othertypes—it turns out that Billey and Lam’s type B vexillary elements arethe same as our vexillary signed permutations, which do not depend ontype. The main goal of this article is to provide alternative characteriza-tions of vexillary signed permutations. In addition to a signed patternavoidance criterion, we give characterizations of vexillary signed per-mutations in terms of essential sets of rank conditions, embeddingsin symmetric groups, and Stanley symmetric functions (the latter asa consequence of the coincidence with Billey-Lam’s type B vexillaryelements).The definition of a vexillary signed permutation w ( τ ), given combi-natorially in §
2, comes with a geometric explanation in terms of degen-eracy loci. Given an odd-rank vector bundle V on a variety X , equippedwith a nondegenerate quadratic form and flags of isotropic subbundles V ⊃ E ⊃ E ⊃ · · · and V ⊃ F ⊃ F ⊃ · · · , a signed permutation w determines a degeneracy locus Ω w ⊆ X , defined by imposing certainrank conditions dim( E p ∩ F q ) ≥ k . Given a triple τ = ( k , p , q ), thevexillary signed permutation w ( τ ) is defined so that the rank condi-tions for the corresponding degeneracy locus are dim( E p i ∩ F q i ) ≥ k i ,for 1 ≤ i ≤ s , and so that w ( τ ) is minimal (in Bruhat order) with thisproperty. The inequalities required on k , p , q guarantee that these rankconditions are feasible: they come from the inclusion of vector spaces( E p i +1 ∩ F q i +1 ) / ( E p i ∩ F q i ) ⊆ E p i +1 /E p i ⊕ F q i +1 /F q i . There is an equivalent interpretation for even-rank vector bundles witha symplectic form. Since a motivating property of our vexillary signed permutations is the flaggedPfaffian formula for Schubert polynomials, they could be called
Pfaffian-vexillary ,when it is necessary to distinguish them from Billey and Lam’s vexillary elements.
EXILLARY SIGNED PERMUTATIONS REVISITED 3
For degeneracy loci corresponding to ordinary permutations, a mini-mal list of non-redundant rank conditions is determined by the essentialset of the permutation [Fu]. The essential set defined in [Fu] consists ofpairs ( p, q ) appearing in rank conditions dim( E p ∩ F q ) ≥ k ; the value of k is determined by a rank function associated to the permutation. Herewe will modify the definitions slightly, for both ordinary and signed per-mutations, to include the value of the rank function. Our essential sets,introduced and studied in [A], consist of certain “basic triples” ( k, p, q );the pairs ( p, q ) will be called essential positions . (Precise definitions arereviewed in § p i , q i ) may be ordered so that p ≥ · · · ≥ p s > q ≥ · · · ≥ q s >
0. Another is given via comparison withsymmetric groups: the definition of the group of signed permutationsleads naturally to an embedding ι : W n ֒ → S n +1 , and remarkably, asigned permutation w is vexillary if and only if ι ( w ) is. (The samestatement applies to another natural embedding ι ′ : W n ֒ → S n .) Wealso give a characterization via signed pattern avoidance , analogous tothe pattern avoidance criterion for S n ; see § Theorem.
Let w be a signed permutation. The following are equiva-lent: (i) w is vexillary, i.e., it is equal to w ( τ ) for some triple τ . (ii) The essential positions of w can be ordered ( p , q ) , . . . , ( p s , q s ) ,so that p ≥ · · · ≥ p s > and q ≥ · · · ≥ q s > . (iii) ι ( w ) is vexillary, as a permutation in S n +1 . (iv) ι ′ ( w ) is vexillary, as a permutation in S n . (v) w avoids the nine signed patterns [2 1] , [3 2 1] , [4 1 2 3] , [4 1 2 3] , [3 4 1 2] , [3 4 1 2] , [2 3 4 1] , [2 3 4 1] , and [3 4 1 2] . The theorem is a consequence of Theorem 2.5 combined with Proposi-tion 3.1 and Remark 3.2.A triple τ determines a strict partition λ ( τ ) of length s , by setting λ k i = p i + q i −
1, and filling in the remaining λ k minimally subjectto the strict decreasing requirement; these partitions index the multi-Schur Pfaffians appearing as degeneracy locus formulas in [AF0]. In § λ ( τ ) computes thelength of w ( τ ) (Corollary 4.5).The pattern avoidance criterion (v) coincides with that of Billey andLam in type B, and as a consequence, we can add another equivalent DAVID ANDERSON AND WILLIAM FULTON condition to the list in the above theorem: writing H w for the Stanleysymmetric function of type B (see, e.g., [BH] or [FK, (6.3)]), we have(vi) H w is equal to a Schur P -function P λ . (In fact, Corollary 5.2 says H w ( τ ) = P λ ( τ ) .) The equivalence of (vi)with (v) is the content of [BL, Theorem 14]. Billey and Lam alsogive pattern avoidance criteria for determining when other variants ofStanley symmetric functions are equal to P - and Q -functions, but ourvexillary signed permutations form a strict subset of the those definedin [BL] for types C and D.Is there a geometric property enjoyed by Billey-Lam’s vexillary el-ements of type C or D? In [AF1], we find “flagged theta-polynomial”formulas for a larger class of signed permutations constructed from gen-eralized triples (so they might be called theta-vexillary ), but they arenot the same as any of Billey-Lam’s vexillary elements. Acknowledgements.
We are grateful to S. Billey for many helpful sug-gestions and comments.
Notation.
We refer to [BB, § . . . , n, . . . , , , , , , . . . , n, . . . , using the bar to denote a negative sign, and we take the natural order onthem, as above. All permutations are finite, in the sense that v ( m ) = m whenever | m | is sufficiently large. We generally write permutations in S n +1 using one-line notation, by listing the values v ( n ) v ( n − · · · v ( n ).A signed permutation is a permutation w with the property that foreach i , w ( ı ) = w ( i ). A signed permutation is in W n if w ( m ) = m for all m > n ; this is a group isomorphic to the hyperoctahedral group, theWeyl group of types B n and C n . When writing signed permutationsin one-line notation, we only list the values on positive integers: w ∈ W n is represented as w (1) w (2) · · · w ( n ). For example, w = 2 1 3is a signed permutation in W , and w (3) = 3 since w (3) = 3. The simple transpositions s , . . . , s n generate W n , where for i >
0, right-multiplication by s i exchanges entries in positions i and i +1, and right-multiplication by s replaces w (1) with w (1). The length of a signedpermutation w is the least number ℓ = ℓ ( w ) such that w = s i · · · s i ℓ .The longest element in W n , denoted w ( n ) ◦ , is 1 2 · · · n , and has length n .The definition of W n presents it as embedded in the symmetric group S n +1 , considering the latter as the group of all permutations of the EXILLARY SIGNED PERMUTATIONS REVISITED 5 integers n, . . . , , . . . , n . This is the odd embedding, and we write ι : W n ֒ → S n +1 for emphasis when a signed permutation is consideredas a full permutation. Specifically, ι sends w = w (1) w (2) · · · w ( n ) tothe permutation w ( n ) · · · w (2) w (1) 0 w (1) w (2) · · · w ( n )in S n +1 . Occasionally, we will refer to the even embedding ι ′ : W n ֒ → S n , defined similarly to ι by considering the even symmetric group aspermutations of {± , . . . , ± n } and omitting the value w (0) = 0.A permutation v has a descent at position i if v ( i ) > v ( i + 1); here i may be any integer. The same definition applies to signed permutations w , but we only consider descents at positions i ≥
0, following theconvention of recording the values of w only on positive integers. Adescent at 0 simply means that w (1) is negative. For example, w = 2 1 3has descents at 0 and 2, while ι ( w ) = 3 1 2 0 2 1 3 has descents at − −
1, 0, and 2. 1.
Diagrams and essential sets
A permutation v ∈ S n +1 can be represented in a (2 n + 1) × (2 n + 1)array of boxes, with rows and columns indexed by { n, . . . , , . . . , n } , byplacing a dot in position ( v ( i ) , i ) for n ≤ i ≤ n ; we refer to this as the permutation matrix of v . The diagram of v is the collection of boxesthat remain after crossing out those weakly south or east of a dot. The rank function of v is defined as(1) r v ( p, q ) = { i ≤ p | v ( i ) ≥ q } , for n ≤ p, q ≤ n . This is equal to the number of dots strictly south andweakly west of the box ( q − , p ) in the permutation matrix of v .The (2 n + 1) numbers r v ( p, q ) determine v , but in fact much lessinformation is required to specify a permutation. A minimal list of rankconditions determining v was given in [Fu], by restricting attention tothe southeast corners of the diagram. More precisely, following theconventions and terminology of [A] (which differ slightly from thoseof [Fu]), a pair ( p, q ) is an essential position if the box ( q − , p ) is asoutheast corner of the diagram of v . The essential set E ss ( v ) is theset of ( k, p, q ) such that ( p, q ) is an essential position and k = r v ( p, q ). By analyzing the possibilities for boxes occurring as southeast corners of thediagram of a permutation, Eriksson and Linusson showed how to reconstruct v from a subset of the essential set [EL]; however, the essential set is minimal in thesense of [Fu, Lemma 3.10]. DAVID ANDERSON AND WILLIAM FULTON
In formulas, a box ( a, b ) is a SE corner of the diagram of v if andonly if v ( b ) > a ≥ v ( b + 1) and(2) v − ( a ) > b ≥ v − ( a + 1) , (2 ′ )so the essential positions are those ( p, q ) such that ( q − , p ) satisfiesEquations (2) and (2 ′ ). This characterizes E ss ( v ) in terms of the de-scents of v , and will be useful later.Analogous diagrams and essential sets for signed permutations weredescribed in [A]. We review the definitions and basic facts briefly here,since essential sets play a role in characterizing vexillary signed permu-tations.For a signed permutation w ∈ W n , the following simple lemma saysthat the essential positions of the corresponding permutation ι ( w ) ∈ S n +1 are “symmetric about the origin” (see Figure 1). Lemma 1.1 ([A, Lemma 1.1]) . For w ∈ W n , the essential set of ι ( w ) ∈ S n +1 possesses the following symmetry: ( k, p, q ) is in E ss ( ι ( w )) if andonly if ( k + p + q − , p + 1 , q + 1) is in E ss ( ι ( w )) . Figure 1.
Diagram and essential set for v = ι ( 2 3 1 ),with circled corners illustrating the symmetry ofLemma 1.1.This implies that half of E ss ( ι ( w )) suffices to determine the signedpermutation w ; we will consider those corners appearing in the first n columns. In general, only a subset of these corners is needed, as shownin [A].As with ordinary permutations, the essential set of a signed permu-tation is defined in terms of its diagram. The permutation matrix of w ∈ W n is a (2 n +1) × n array of boxes, with rows labelled n, . . . , , . . . , n and columns labelled n, . . . ,
1. A dot is placed in the boxes ( w ( i ) , ı ) for EXILLARY SIGNED PERMUTATIONS REVISITED 7 × × × × × × × × × × × ×× × × × × × × × × × ×× × × × × × × × × ×× × × × × × × × ×× × × × × × × ×× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××
13 4 5 8 w = 1 2 7 11 6 8 5 3 12 10 9 4 E ss ( w ) = { (1 , , , (3 , , , (4 , , , (5 , , , (8 , , } ℓ ( w ) = 103 Figure 2.
Diagram and essential set of a signed permu-tation. (This is also the vexillary signed permutation forthe triple τ = ( 1 3 4 5 8 , ,
12 9 8 8 5 ).)1 ≤ i ≤ n . Additionally, for each dot, the box in the same column butopposite row is marked with an × , as is each box to the right of thisone. (That is, an × is placed in each box ( a, b ) such that a = w ( i ) forsome i ≤ b .) The extended diagram of w is the set e D w of boxes thatremain after crossing out those south or east of a dot in the permuta-tion matrix; the diagram D w is the subset of e D w not marked by an × . The placement of the × ’s is closely related to the parametrizationof Schubert cells described in [FP], and the boxes of D w are in naturalbijection with the inversions of w . In particular, the number of boxesin D w is equal to the length of w . See Figure 2 for an illustration. DAVID ANDERSON AND WILLIAM FULTON
The rank function of w is defined as(3) r w ( p, q ) = { i ≥ p | w ( i ) ≤ q } . Since w ( ı ) = w ( i ), this is equivalent to(4) r w ( p, q ) = { i ≤ p | w ( i ) ≥ q } . The essential set of a signed permutation w is the set of ( k, p, q )such that ( q − , p ) is a SE corner of the extended diagram e D w and k = r w ( p, q ), with two exceptions. First, if p = 1 and q < k, p, q ) is not in E ss ( w ). Second, when p > q >
0, ( k, p, q ) is not in E ss ( w ) if there is another SE cornerin box ( q, p ), and k = r w ( p, q ) = r w ( p, q + 1) − q + 1.The exceptions are easy to understand in the context of linear alge-bra; see [A] for more explanation. The first one comes from Lemma 1.1:such corners are artifacts of restricting the permutation matrix to n columns, and they do not appear as corners of the diagram of ι ( w ).The second, more complicated exception never applies to the vexillarysigned permutations to be defined in the next section. As we will seein Lemma 2.4, when w is vexillary, no SE corners of e D w lie above themiddle row, except possibly in the rightmost column.As with ordinary permutations, a signed permutation is determinedby its essential set: w is the minimal element (in Bruhat order on W n )such that r w ( p, q ) ≥ k for all ( k, p, q ) ∈ E ss ( w ). Furthermore, E ss ( w ) isthe smallest set with this property—choosing any ( k , p , q ) ∈ E ss ( w ),there exists a w ′ = w such that r w ′ ( p , q ) < k and r w ′ ( p, q ) ≥ k forall ( k, p, q ) ∈ E ss ( w ) r { ( k , p , q ) } . (This the content of [A, Theo-rem 2.3].)2. Triples and vexillary signed permutations
As defined in the introduction, triple consists of three s -tuples ofpositive integers τ = ( k , p , q ), with k = (0 < k < · · · < k s ) , p = ( p ≥ · · · ≥ p s > , q = ( q ≥ · · · ≥ q s > , satisfying(*) p i − p i +1 + q i − q i +1 ≥ k i +1 − k i for 1 ≤ i ≤ s −
1. The triple is essential if the inequalities (*) are strict.(These inequalities ensure that the rank conditions dim( E p i ∩ F q i ) ≥ k i are feasible, and strict inequalities ensure they are independent.) Each EXILLARY SIGNED PERMUTATIONS REVISITED 9 triple reduces to a unique essential triple, by successively removing each( k i , p i , q i ) such that equality holds in (*). Two triples are equivalent if they reduce to the same essential triple.Given a triple, one forms a signed permutation w ( τ ) as follows.(1) Starting in the p position, place k consecutive negative en-tries, in increasing order, ending with q . Mark these numbersas “used”.(i) For 1 < i ≤ s , starting in the p i position, or the next avail-able position to the right, fill the next available k i − k i − po-sitions with negative entries chosen consecutively from the un-used numbers, ending with at most q i . (Note: the sequencemay have to “jump” over previously placed sequences.) (s+1) Fill the remaining available positions with the unused positivenumbers, in increasing order. Definition 2.1.
A signed permutation w ∈ W n is vexillary if w = w ( τ ) for some triple τ = ( k , p , q ). By convention, the “empty” tripleis a triple, so the identity element is vexillary.A triple also defines a strict partition λ ( τ ), by setting λ k i = p i + q i − λ k = p i + q i − k i − k if k i − < k ≤ k i .)For example, with τ = ( 1 3 4 5 8 , ,
12 9 8 8 5 ), the sixsteps in forming w ( τ ) produce w = · · · · · · · · ,w = · · · · · · · ·
10 9 ,w = · · · · · · ·
12 10 9 ,w = · · · · · ·
12 10 9 ,w = · · ·
12 10 9 ,w =
12 10 9 . The corresponding partition is λ = (20 , , , , , , , It is useful to break each step in the construction into sub-steps. In Step (i),first pick out the k i − k i − largest unused entries less than or equal to q i and placethem in a “bin”. Second, go to position p i : If this position is available, take thesmallest entry from the bin, place it here, and move to position p i + 1; if position p i is unavailable, just move to position p i + 1. Repeat the second sub-step, startingat position p i + 1. Finish when the bin is empty. and the same strict permutation, so we will generally assume triplesare essential. Lemma 2.2.
Let w = w ( τ ) , for an essential triple τ = ( k , p , q ) . Thedescents of w are at the positions p i − . In fact, for each i , we have w ( p i − > q i ≥ w ( p i ) , and there are no other descents.Proof. In Step (1), no descents are created, unless p = 1, in which casethe permutation has a single descent at 0.Now for 1 < i ≤ s , consider the situation before Step (i) in con-structing w ( τ ). Assume inductively that for j < i , there is a descentat position p j − w ( p i − > q i ≥ w ( p i ), and there are no other descents.In carrying out Step (i), we place negative entries in consecutivevacant positions, from left to right, starting at position p i (or the nextvacant position to the right, if p i = p i − ). We consider “sub-steps”of Step (i), where we are placing an entry at position p ≥ p i , anddistinguish three cases.First, suppose we are at position p , with p < p i − −
1. In this case,the previous entry placed in Step (i) (if any) was placed at position p −
1, so we did not create a descent at p −
1. Position p + 1 is stillvacant, so no new descents are created.To illustrate, take τ = ( 1 3 4 5 8 , ,
12 9 8 8 5 ), as in theexample above. In Step (3), we place 8 without creating a descent: w = · · · · · · ·
12 10 9 . Next, suppose we are at position p = p i − −
1. This means p i − − p i ≤ k i − k i − , so let β = ( k i − k i − ) − ( p i − − p i ) be the number of entriesremaining to be placed in Step (i), after placing the current one. Theinequality (*) in the triple condition implies q i + β < q i − , which inturn means that the entries used in previous steps are all strictly lessthan q i + β ; therefore the β + 1 entries q i + β, q i + β − , . . . , q i are allavailable to be placed. It follows that w ( p i − −
1) = q i + β , whichcreates a descent satisfying the claim of the lemma. Again, we did notcreate a descent at position p −
1, for the same reason as in the previouscase.Continuing the running example, the first entry placed in Step (5) isthe 7, creating a descent at position 3: w = · · · · ·
12 10 9 . (In this case, we had q = 5 and β = 2.)Finally, suppose we are at position p = p i − . Set β = ( k i − k i − ) − ( p i − − p i ) −
1, so β ≥
0; this is the number of entries to be placed after
EXILLARY SIGNED PERMUTATIONS REVISITED 11 the current one. Now the essential triple condition implies q i + β + 1 1, for j < i − p = p − p . The entries to be placed are 6 , 5. In placing the6 in the next vacant position—position 5—we do not create a descentat the filled position to its left, but we do create a descent at position5, since position 6 is already filled: w = · · · 12 10 9 . In placing the 5, there is no descent created, since the position to itsright is vacant.At Step (s+1), the same reasoning as in the last case consideredshows that descents are created at those p j − (cid:3) Given a triple τ = ( k , p , q ), the dual triple is τ ∗ = ( k , q , p ). Lemma 2.3. We have w ( τ ∗ ) = w ( τ ) − .Proof. Suppose w ∈ W n , and consider ι ( w ) as a bijective map fromthe set { n, . . . , , . . . , n } to itself. To determine ι ( w ), it is enoughto know its values on any set of integers whose absolute values are { , . . . , n } . The construction of w translates as follows. Let a (1) = { p , p + 1 , . . . , p + k − } , let a (2) be the set of k − k consecutiveintegers in { , . . . , n } r a (1) starting from the p th element, and de-fine sets a (3) , . . . , a ( s ) similarly. Let a ( s + 1) be the complement of a (1) , . . . , a ( s ), so the sets a (1) , . . . , a ( s + 1) partition { , . . . , n } . Define b (1) , . . . , b ( s + 1) in the same way, using q in place of p . Also write a ( i ) and b ( i ) for the corresponding sets of negative integers. Now ι ( w )maps a ( i ) to b ( i ) for 1 ≤ i ≤ s , and it maps a ( s + 1) to b ( s + 1). Itfollows that ι ( w ) − maps b ( i ) to a ( i ), or equivalently, b ( i ) to a ( i ), andmaps b ( s + 1) to a ( s + 1). In other words, the inverse is obtained byswitching the roles of a and b , so the lemma is proved. (cid:3) Lemma 2.4. (1) Let w = w ( τ ) be a vexillary signed permutation, foran essential triple τ = ( k , p , q ) . Then the SE corners of the diagram of ι ( w ) are the boxes ( q i − , p i ) , together with their reflections ( q i , p i − .(In particular, no box ( a, b ) with a, b < occurs as a SE corner.) (2) k i is the number of dots strictly south and weakly west of the i thSE corner in the diagram (in position ( q i − , p i ) ).Proof. By Lemma 1.1, it suffices to show that the first s corners ofthe diagram of ι ( w ) (ordered SW to NE) are as claimed. For p > w has a descent at position p − ι ( w ) hasdescents at positions p − p . By Lemma 2.2, the descents of ι ( w )are at positions p i − p i , and the inequalities (2) ι ( w )( p i ) > q i − ≥ ι ( w )( p i − w ( p i − ≥ q i − > w ( p i ) , using Lemma 2.2 again.A similar argument establishes the inequalities (2 ′ ): ι ( w ) − ( q i − > p i ≥ ι ( w ) − ( q i )(Swap p and q , and apply Lemma 2.2 to w − = w ( k , q , p ).)Part (2) is easy from the construction: At the end of Step (i), all theentries to the right of p i are at most q i , and there are k i of them. AfterStep (i), any entry placed to the right of p i is greater than q i . (cid:3) Theorem 2.5. For w ∈ W n , the following are equivalent: (i) The signed permutation w is vexillary. (ii) The essential positions ( p i , q i ) of w can be ordered so that p ≥· · · ≥ p s > and q ≥ · · · ≥ q s > . (iii) The permutation ι ( w ) is vexillary (as an element of S n +1 ).Proof. We will show (i) ⇒ (ii) ⇒ (iii) ⇒ (i). The implication (i) ⇒ (ii)is immediate from Lemma 2.4, and (ii) ⇒ (iii) follows from Lemma 1.1.It remains to show (iii) ⇒ (i).Suppose ι ( w ) is vexillary, and recall that this is equivalent to re-quiring that the SE corners of its diagram proceed from southwest tonortheast [Fu, Remark 9.17]. Take those SE corners ( a, b ) such that b < 0; Lemma 1.1 implies a ≥ 0. Let s = ⌈ E ss ( ι ( w ))2 ⌉ be the num-ber of such boxes. Reading from SW to NE, label them ( a i , b i ), with1 ≤ i ≤ s . Set p i = − b i and q i = a i + 1, and let k i be the number ofdots strictly south and weakly west of ( a i , b i ).We claim that τ = ( k , p , q ) is an essential triple. Indeed, looking atthe extended diagram e D w , the fact that the boxes ( q i − , p i ) are SE cor-ners implies the inequalities (*). It follows that w is equal to w ( τ ), sincea permutation is determined by its essential set [Fu, Lemma 3.10(b)],[A, Theorem 2.3]. (cid:3) EXILLARY SIGNED PERMUTATIONS REVISITED 13 Pattern avoidance Given a permutation π in S m , a permutation v contains the pattern π if, when written in one-line notation, there is an m -element subsequenceof v which is in the same relative order as π ; otherwise v avoids π . Thereis a similar notion for signed permutations and signed patterns. Givena signed pattern π = π (1) π (2) · · · π ( m ) in W m , a signed permutation w contains π if there is a subsequence w ( i ) · · · w ( i m ) such that the signsof w ( i j ) and π ( j ) are the same for all j , and also the absolute values ofthe subsequence are in the same relative order as the absolute valuesof π . Otherwise w avoids π . (See, e.g., [BL, Definition 6].)For example, 5 1 3 2 4 contains the pattern [3 2 1], as the subsequence5 3 2, but 5 1 2 3 4 avoids [3 2 1]. Proposition 3.1. A signed permutation w is vexillary if and only if w avoids the signed patterns [2 1] , [3 2 1] , [4 1 2 3] , [4 1 2 3] , [3 4 1 2] , [3 4 1 2] , [2 3 4 1] , [2 3 4 1] , and [3 4 1 2] .Proof. We will use the characterization of Theorem 2.5, and show thata signed permutation w avoids these nine patterns if and only if ι ( w )is vexillary in S n +1 . Recall that a permutation is vexillary if and onlyif it avoids the pattern [2 1 4 3]. When embedded by ι , the nine signedpatterns listed in (ii) all contain [2 1 4 3], so if ι ( w ) avoids [2 1 4 3],then w must avoid these signed patterns; this proves the “ ⇒ ” direction.For the “ ⇐ ” implication, we claim that if ι ( w ) contains the pattern[2 1 4 3], then w contains one of the nine signed patterns listed in theProposition. To check this, we break the ways [2 1 4 3] can appear in ι ( w ) into cases, and observe that in each case, one of the listed patternsappears.For keeping track of the cases, we introduce some temporary nota-tion. An instance of the pattern [2 1 4 3] is witnessed by ι ( w )( b ) <ι ( w )( a ) < ι ( w )( d ) < ι ( w )( c ) for some a < b < c < d . We record whichof a, b, c, d are negative by placing a vertical bar in the word [2 1 4 3],and which of the corresponding values of ι ( w ) are negative by placingunderlines. For example, if w = 2 3 5 4 1, so ι ( w ) = 1 4 5 3 2 0 2 3 5 4 1 , then an instance of [2 1 4 3] occurs as [2 | | | | | α < β < γ , so an occurrence of [ | is a subsequence β γ x α . This is the signed pattern [2 3 4 1] if x > γ ,it is [3 4 1 2] if x < α , and it contains [3 2 1] if α < x < γ . Finally, anoccurrence of [ | | | | a < b < c , the case [2 | β γ x α is the signed pattern[2 3 4 1] if x > γ ; it is [3 4 1 2] if x < α , and it contains [3 2 1] if α < x < γ . The subsequences γ β x α and γ x β α contain [3 2 1]. Thesubsequence γ x α β is the signed pattern [4 1 2 3] if x < α , it contains[3 2 1] if α < x < β , and it contains [2 1] if x > β . Similarly, dependingon the position of the “2”, the case [2 | | | | | | (cid:3) Remark 3.2. An identical analysis shows that w avoids the same ninesigned patterns if and only if ι ′ ( w ) ∈ S n avoids [2 1 4 3]; therefore w is vexillary if and only if ι ′ ( w ) is vexillary.The pattern avoidance criterion for ordinary permutations allowedJ. West to enumerate the vexillary permutations in S n [We]. His proofuses a bijection between permutations avoiding [2 1 4 3] (vexillarypermutations) and those that avoid [4 3 2 1]. One might expect asimilar enumeration of vexillary signed permutations. In fact, for theeven embedding ι ′ : W n ֒ → S n (which omits “ w (0) = 0”), Egge showedthat the number of w ∈ W n such that ι ′ ( w ) avoids [4 3 2 1] is equal to V n := n X k =0 (cid:18) nk (cid:19) C k , where C k = k +1 (cid:0) kk (cid:1) is the k th Catalan number [Eg]. Together withcomputer verification up to n = 7, this suggests the conjecture that V n is also the number of vexillary signed permutations in W n . West’s bi-jection does not preserve the subgroup of signed permutations, though,so the proof is not immediately clear.4. Labelled Young diagrams There is an alternative way to encode vexillary signed permutations,useful for determining when right-multiplication by a simple reflectiontakes one vexillary signed permutation to another. Identifying a strictpartition λ with its shifted Young diagram, a labelled Young dia-gram of shape λ is an assignment of integers to the SE corners, weakly EXILLARY SIGNED PERMUTATIONS REVISITED 15 11 20 Figure 3. Labelled Young diagrams associated to τ =( 2 3 , , τ ∗ = ( 2 3 , , m , . . . , m s in rows k , . . . , k s satisfy 0 ≤ m i < λ k i and m i − m i +1 ≤ λ k i − λ k i +1 . (Note that λ k i − λ k i +1 + 1 is the number of boxes in the rim-hook connecting thecorners labelled m i and m i +1 .)Labelled Young diagrams are in bijection with essential triples: given τ = ( k , p , q ), the shifted Young diagram for λ ( τ ) has SE corners inrows k , . . . , k s , and one forms a labelled Young diagram of this shapeby placing integers m i = p i − w = w ( τ ), and let Y be the corresponding labelled Young dia-gram of shape λ ( τ ). By Lemma 2.2, the descents of w are the cornerlabels of Y , i.e., ℓ ( ws m ) = ℓ ( w ) − m is a label.A corner label m in Y is removable if • it appears in Y exactly once, and • when m = 0, the row in which it appears contains a single box.Given a removable label m , one can remove the box containing it toform a new labelled Young diagram Y r m , whose corners are labelledaccording to the following four rules.(i) If a corner of Y r m is also a corner of Y , its label is the same.(ii) If removing m produces a new corner one box to the left, labelthat corner m − m produces a new corner one box above, label thatcorner m + 1.(iv) If removing m produces two new corners, label the one to theleft m − m + 1.These are the only possibilities for removing a corner from the Youngdiagram. Examples are shown in Figure 4.If w is the vexillary signed permutation corresponding to a labelledYoung diagram Y , then ws m is vexillary when m is removable: the r −→ r −→ r −→ r −→ Figure 4. Removing corner labels.reader may verify that Y r m is its corresponding labelled Young dia-gram. The next theorem says the converse also holds. Theorem 4.1. Let Y be the labelled Young diagram corresponding to avexillary signed permutation w . Then ws m is vexillary of length ℓ ( w ) − if and only if m is a removable label of Y .Proof. Let τ = ( k , p , q ) be the corresponding essential triple. Usingthe pattern avoidance criterion for vexillarity, we will show that if alabel m is not removable, then ws m is not vexillary.First suppose m occurs more than once as a label in Y , or equiva-lently, the index p = m + 1 is repeated in the sequence p . From theconstruction of w ( τ ), the repetition of p means there is a gap in theincreasing sequence of negative integers starting at position p in w . Inother words, there is a “partial pattern” [3 1] in w , with the 3 occurringat position p .If m = 0, then the gap must be filled by a positive integer occurringto the right. That is, the pattern [3 1 2] occurs in w , with the “3in position 1. We find that ws contains the pattern [3 1 2], so inparticular it contains [2 1] and is not vexillary. EXILLARY SIGNED PERMUTATIONS REVISITED 17 If m > 0, and the entry of w in position m = p − p − p ) , [3 1 4 2] (with “1” at position p − p ) , [2 3 1] (with “2” at position p − p ) , [2 4 3 1] (with “4” at position p − p ) , or[2 4 3 1] (with “4” at position p − p ) . If the entry in position p − p − p . The gap can befilled to yield one of three patterns:[1 4 2 3] , [3 1 4 2] , or [3 1 4 2] . In each case, right-multiplication by s m yields a forbidden pattern,verifying that ws m is not vexillary.Finally, suppose m = 0 occurs as a label in a row with more than onebox. We may assume the label is not repeated, so this must be the lastrow of Y , i.e., p s = 1 and q s > 1. From the construction, this meansthe positive integer 1 occurs to the right of position 1 in w . Applying s therefore results in the forbidden pattern [2 1]. (cid:3) By reversing the rules for removing a corner, one obtains rules foradding a box to a labelled Young diagram. Suppose Y has labels m i for corners in rows k i , and fix an index j such that m j − m j +1 > 1. (Toinclude extreme cases j = 0 and j = s , we allow 0 ≤ j ≤ s and use theconventions k = 0, m = + ∞ , λ = + ∞ , k s +1 = k s + 1, m s +1 = − λ k s +1 = 0.) A new labelled Young diagram Y ∪ m is defined asfollows, for certain m to be specified.(i) If k j +1 − k j > λ k j − λ k j +1 > k j +1 − k j + 1, then Y ∪ m isdefined for any m satisfying m j > m > m j +1 ,m − m j +1 ≤ k j +1 − k j , and m j − m ≤ λ k j − λ k j +1 + k j − k j +1 , by creating a new corner labelled m in row k j + 1 and leavingall other labels unchanged.(ii) If k j +1 − k j = 1 and λ k j − λ k j +1 > 2, then Y ∪ m is defined for m = m j +1 + 1, by placing a new box labelled m in row k j + 1and erasing the old label m j +1 . (iii) If k j +1 − k j > λ k j − λ k j +1 = k j +1 − k j + 1, then Y ∪ m is defined for m = m j − 1, by placing a new box labelled m inrow k j + 1 and erasing the old label m j .(iv) If k j +1 − k j = 1 and λ k j − λ k j +1 = 2, then Y ∪ m is defined for m = m j +1 + 1 = m j − 1, by placing a new box labelled m inrow k j + 1 and erasing the old labels m j and m j +1 .An integer m is insertable in a labelled Young diagram Y if there isa j such that m j > m > m j +1 and one of the above four cases holds. Corollary 4.2. Suppose w is vexillary, with labelled Young diagram Y .Then ws m is vexillary of length ℓ ( w ) + 1 if and only if m is insertablein Y , in which case its labelled Young diagram is Y ∪ m . For a labelled Young diagram Y with labels m i in rows k i , set l i = λ k i − m i − 1, and define n ( Y ) = max { m i + k i , l i + k i } ≤ i ≤ s . Corollary 4.3. Let Y be a labelled Young diagram with correspondingvexillary signed permutation w . If n ≥ n ( Y ) , then there is a sequence ofvexillary signed permutations w = w (0) , w (1) , . . . , w ( r ) = w ( n ) ◦ , such thatfor each i there is an m with w ( i +1) = w ( i ) s m and ℓ ( w ( i +1) ) = ℓ ( w ( i ) ) + 1 .Proof. Unless Y has shape λ = (2 n − , n − , . . . , 1) with labels n − , n − , . . . , w ( n ) ◦ ),there is an insertable m with 0 ≤ m ≤ n − 1. Indeed, suppose n ≥ n ( Y ),but no such insertable m exists. Then rules (i) and (ii) imply m = n − m = n − n th row. A diagram with these labels has n ( Y ) ≤ n onlyif it is the one corresponding to w ( n ) ◦ (in which case n ( Y ) = n ). (cid:3) The inequalities on m i imply that the labels can be replaced by l i = λ k i − m i − dual labelled Young diagram Y ∗ . If Y corresponds to the triple τ , then Y ∗ corresponds to the dual triple τ ∗ , so the corresponding vexillary signed permutations are inverses.The analogous results for left-multiplication by s m follow from theseobservations. Corollary 4.4. Let Y be a labelled Young diagram corresponding to avexillary signed permutation w . Then s m w is vexillary of length ℓ ( w ) − if and only if m is a removable label of Y ∗ . Similarly, s m w is vexillaryof length ℓ ( w ) + 1 if and only if m is insertable in Y ∗ .If n ≥ n ( Y ) , then there is a sequence of vexillary signed permutations w = w (0) , w (1) , . . . , w ( r ) = w ( n ) ◦ , such that for each i there is an m with w ( i +1) = s m w ( i ) and ℓ ( w ( i +1) ) = ℓ ( w ( i ) ) + 1 . EXILLARY SIGNED PERMUTATIONS REVISITED 19 Inserting a label is a more flexible operation than removing a label.Starting from a given vexillary permutation w , there need not be adescending chain of vexillary permutations w ( i ) ending in w ( r ) = id,with w ( i +1) = w ( i ) s m and ℓ ( w ( i +1) ) = ℓ ( w ( i ) ) − 1. For instance, taking τ = (2 3 , , m such that ℓ ( ws m ) = ℓ ( w ) − m = 1, but this is not a removable label.However, at least one of Y or Y ∗ always has a removable label. Let-ting τ be the corresponding triple, the next statement is proved byinduction on the length of w ( τ ), removing a label from either Y or Y ∗ . Corollary 4.5. For w = w ( τ ) , we have ℓ ( w ) = | λ ( τ ) | . (This can also be proved directly from the construction of w ( τ ) bycounting inversions.) 5. Transitions A signed permutation w is maximal grassmannian if its only descentis at 0. A maximal grassmannian signed permutation corresponds to astrict partition λ , by recording the absolute values of the barred entries;for example, w = 4 2 1 3 corresponds to λ = (4 , , λ and all corners la-belled 0. For a strict partition λ , we will write w λ for the correspondingmaximal grassmannian signed permutation. Transitions provide a way of reducing arbitrary signed permutationsto maximal grassmannian ones. They were used by Billey to studySchubert polynomials and Stanley symmetric functions [Bi].For i < j , let t ij be the transposition exchanging positions i and j , and for i ≤ j let s ij exchange i and . (Thus t ij is the reflectionin the hyperplane defined by e i − e j , and s ij is the reflection in thehyperplane defined by e i + e j .) For any signed permutation w , let m be the last descent, and let j be the largest index greater than m suchthat w ( m ) > w ( j ). A transition of w is a signed permutation w − ,of the same length as w , such that w − = wt mj t im for some i < m or w − = wt mj s im for any i .Definitions and properties of the type B Stanley symmetric function H w may be found in [BH] or [FK]. Here we need two properties, from[Bi]. First, for a strict partition λ , we have(5) H w λ = P λ , where the latter is the Schur P -function. Second, for any w , with m and j defined as above, there is a recursive formula(6) H w = ( 2 H wt mj s mm + ) X w − = wt mj s mm H w − , the sum over transitions of w , with the first term appearing when ℓ ( wt mj s mm ) = ℓ ( w ) (i.e., when this is also a transition). Lemma 5.1. Let Y be a labelled Young diagram, and assume the largestlabel m is greater than . Suppose m = m = · · · = m r > m r +1 , and let Y − be the result of replacing the r th corner label m r by m − . Let w and w − be the corresponding vexillary signed permutations. Then w − is the unique transition of w .Proof. Let τ = ( k , p , q ) be the triple corresponding to Y . From theconstruction, m = p − · · · = p r − 1, and j = m + k r . Notethat w ( j ) = q r . The transposition t mj swaps w ( m ) and w ( j ); since thesequence w ( p ) , . . . , w ( j ) is increasing, ℓ ( wt mj ) = ℓ ( w ) − m that is less than q r , let i < m be the largest such index. Then w − = wt mj t im is a transition of w , and one checks that its labelled Young diagram is Y − . Furthermore,for any i < i ′ < m , right-multiplication by t i ′ m decreases the length of wt mj , and for any 0 < i ′ < i , right-multiplication by t i ′ m increases thelength of wt mj by at least 2; similarly, ℓ ( wt mj s i ′ m ) = ℓ ( w ) for any i ′ .If there is no entry to the left of m less than q r , let i > m be thesmallest index such that w ( i ) > q r . Then w − = wt mj s mi is a transitionof w , with labelled Young diagram Y − . One checks as before that thisis the only transition. (cid:3) Combining Equations (5) and (6) with Lemma 5.1 yields a formulafor the Stanley symmetric function of a vexillary signed permutation. Corollary 5.2. The Stanley symmetric function H w ( τ ) is equal to theSchur P -function P λ ( τ ) . It follows that for a vexillary signed permutation w = w ( τ ), thepartition λ B ( w ) defined in [BL, § 5] is equal to our λ ( τ ). References [A] D. Anderson, “Diagrams and essential sets for signed permutations,”preprint, arXiv:1612.08670 [math.CO] .[AF0] D. Anderson and W. Fulton, “Degeneracy loci, Pfaffians, and vexillarysigned permutations in types B, C, and D,” preprint, arXiv:1210.2066[math.AG] . EXILLARY SIGNED PERMUTATIONS REVISITED 21 [AF1] D. Anderson and W. Fulton, “Chern class formulas for classical-type de-generacy loci,” preprint, arXiv:1504.03615v2 [math.AG] .[Bi] S. Billey, “Transition equations for isotropic flag manifolds,” Discrete Math. (1998), 69–84.[BH] S. Billey and M. Haiman, “Schubert polynomials for the classical groups,” J. Amer. Math. Soc. (1995), no. 2, 443–482.[BL] S. Billey and T. K. Lam, “Vexillary elements in the hyperoctahedralgroup,” J. Algebraic Combin. (1998), no. 2, 139–152.[BB] A. Bj¨orner and F. Brenti, Combinatorics of Coxeter groups , Springer, 2005.[Eg] E. Egge, “Enumerating rc -invariant permutations with no long decreasingsubsequences,” Ann. Combin. (2010), no. 1, 85–101.[EL] K. Eriksson and S. Linusson, “Combinatorics of Fulton’s essential set,” Duke Math. J. (1996), no. 1, 61–76.[FK] S. Fomin and A. N. Kirillov, “Combinatorial B n -analogues of Schubertpolynomials”, Trans. Amer. Math. Soc. (1996), 3591–3620.[Fu] W. Fulton, “Flags, Schubert polynomials, degeneracy loci, and determi-nantal formulas,” Duke Math. J. (1992), no. 3, 381–420.[FP] W. Fulton and P. Pragacz, Schubert Varieties and Degeneracy Loci ,Springer, 1998.[IMN] T. Ikeda, L. Mihalcea, and H. Naruse, “Double Schubert polynomials forthe classical groups,” Adv. Math. (2011), no. 1, 840–886.[Ka] M. Kazarian, “On Lagrange and symmetric degeneracy loci,” preprint,Arnold Seminar (2000).[LS1] A. Lascoux and M.-P. Sch¨utzenberger, “Polynˆomes de Schubert,” C.R. Acad. Sci. Paris S´er. I Math. (1982), 447–450.[LS2] A. Lascoux and M.-P. Sch¨utzenberger, “Schubert polynomials and theLittlewood-Richardson rule,” Letters in Mathematical Physics (1985),111–124.[Mac] I. G. Macdonald, Notes on Schubert Polynomials , Publ. LACIM 6, Univ. deQu´ebec `a Montr´eal, Montr´eal, 1991.[We] J. West, “Generating trees and the Catalan and Schr¨oder numbers,” Dis-crete Math. (1995), no. 1-3, 247–262. Department of Mathematics, The Ohio State University, Columbus,OH 43210 E-mail address : [email protected] Department of Mathematics, University of Michigan, Ann Arbor,Michigan 48109-1043, U.S.A. E-mail address ::