Nil Bohr 0 -sets, Poincaré recurrence and generalized polynomials
aa r X i v : . [ m a t h . D S ] S e p NIL BOHR -SETS, POINCAR ´E RECURRENCE ANDGENERALIZED POLYNOMIALS WEN HUANG, SONG SHAO, AND XIANGDONG YE
Abstract.
The problem which can be viewed as the higher order version of anold question concerning Bohr sets is investigated: for any d ∈ N does the collectionof { n ∈ Z : S ∩ ( S − n ) ∩ . . . ∩ ( S − dn ) = ∅} with S syndetic coincide with thatof Nil d Bohr -sets?In this paper it is proved that Nil d Bohr -sets could be characterized via gen-eralized polynomials, and applying this result one side of the problem could beanswered affirmatively: for any Nil d Bohr -set A , there exists a syndetic set S such that A ⊃ { n ∈ Z : S ∩ ( S − n ) ∩ . . . ∩ ( S − dn ) = ∅} . Note that other side ofthe problem can be deduced from some result by Bergelson-Host-Kra if modulo aset with zero density. As applications it is shown that the two collections coincidedynamically, i.e. both of them can be used to characterize higher order almostautomorphic points. Introduction
Combinatorial number theory attracts a lot of attention. In such a theory, prob-lems concerning Bohr sets are extensively studied, and have a long history which atleast could be traced back to the work of Veech in 1968 [29]. Bohr sets are funda-mentally abelian in nature. Nowadays it has become apparent that a higher ordernon-abelian Fourier analysis plays a role both in combinatorial number theory andergodic theory. Related to this, a higher-order version of Bohr sets, namely Nil d Bohr -sets, was introduced in [18]. For the recent results obtained by Katznelson,Bergelson-Furstenberg-Weiss and Host-Kra see [23, 3, 18].1.1. Nil-Bohr sets.
There are several equivalent definitions for Bohr sets. Here isthe one easy to understand: a subset A ⊆ Z is a Bohr set if there exist m ∈ N , α ∈ T m , and an open set U ⊆ T m such that { n ∈ Z : nα ∈ U } is contained in A ;the set A is a Bohr -set if additionally 0 ∈ U .It is not hard to see that if ( X, T ) is a minimal equicontinuous system, x ∈ X and U is a neighborhood of x , then N ( x, U ) =: { n ∈ Z : T n x ∈ U } contains S − S =: { a − b : a, b ∈ S } with S syndetic, i.e. with a bounded gap. An oldquestion concerning Bohr sets is Date : August 24, 2011.2000
Mathematics Subject Classification.
Primary: 37B05, 22E25, 05B10.
Key words and phrases.
Nilpotent Lie group, nilsystem, Bohr set, Poincar´e recurrence, andgeneralized polynomials.Huang is supported by Fok Ying Tung Education Foundation, Shao is supported by NNSF ofChina (10871186) and Program for New Century Excellent Talents in University, and Huang+Yeare supported by NNSF of China (11071231). sets Problem A-I:
Let S be a syndetic set of Z , is S − S a Bohr -set?That is, are the common differences of arithmetic progressions with length 2appeared in a syndetic set a Bohr set? Veech showed that it is at least “almost”true [29]. That is, given a syndetic set S ⊆ Z , there is some subset N with densityzero such that ( S − S )∆ N is a Bohr -set.A subset A ⊆ Z is a Nil d Bohr -set if there exist a d -step nilsystem ( X, T ), x ∈ X and an open set U ⊆ X containing x such that N ( x , U ) =: { n ∈ Z : T n x ∈ U } is contained in A . Denote by F d, the family consisting of all Nil d Bohr -sets. We can now formulate a higher order form of Problem A-I. We notethat { n ∈ Z : S ∩ ( S − n ) ∩ . . . ∩ ( S − dn ) = ∅} can be viewed as the commondifferences of arithmetic progressions with length d + 1 appeared in the subset S .In fact, S ∩ ( S − n ) ∩ . . . ∩ ( S − dn ) = ∅ if and only if there is m ∈ S with m, m + n, . . . , m + dn ∈ S . Problem B-I: [Higher order form of Problem A-I] Let d ∈ N .(1) For any Nil d Bohr -set A , is it true that there is a syndetic subset S of Z with A ⊃ { n ∈ Z : S ∩ ( S − n ) ∩ . . . ∩ ( S − dn ) = ∅} ? (2) For any syndetic set S , is { n ∈ Z : S ∩ ( S − n ) ∩ . . . ∩ ( S − dn ) = ∅} a Nil d Bohr -set? Dynamical version of the higher order Bohr problem.
Sometimes com-binatorial questions can be translated into dynamical ones by the Furstenberg cor-respondence principle, see Section 2.4. Using this principle, it can be shown thatProblem A-I is equivalent to the following version:
Problem A-II:
For any minimal system ( X, T ) and any nonempty open set U ⊂ X ,is the set { n ∈ Z : U ∩ T − n U = ∅} a Bohr -set? Similarly, Problem B-I has its dynamical version:
Problem B-II: [Dynamical version of Problem B-I] Let d ∈ N .(1) For any Nil d Bohr -set A , it is true that there are a minimal system ( X, T ) and a non-empty open subset U of X with A ⊃ { n ∈ Z : U ∩ T − n U ∩ . . . ∩ T − dn U = ∅} ?(2) For any minimal system ( X, T ) and any open non-empty U ⊂ X , is it truethat { n ∈ Z : U ∩ T − n U ∩ . . . ∩ T − dn U = ∅} a Nil d Bohr -set? It follows from some result by Bergelson-Host-Kra in [4] that Problem B-II(2) hasa positive answer if ignoring a set with zero density. In fact, the authors [4] showed:Let ( X, X , µ, T ) be an ergodic system and d ∈ N , then for all A ∈ X with µ ( A ) > I = { n ∈ Z : µ ( A ∩ T − n A ∩ . . . ∩ T − dn A ) > } is almost a Nil d Bohr -set,i.e. there is some subset N with density zero such that I ∆ N is a Nil d Bohr -set. A collection F of subsets of Z (or N ) is a family if it is hereditary upward, i.e. F ⊆ F and F ∈ F imply F ∈ F . Any nonempty collection A of subsets of Z generates a family F ( A ) := { F ⊆ Z : F ⊃ A for some A ∈ A} . . Huang, S. Shao and X.D. Ye 3 Main results.
We will show that Problem B-II(1) has an affirmative answer.Namely, we will show
Theorem A : Let d ∈ N . If A ⊆ Z is a Nil d Bohr -set, then there exist a minimal d -step nilsystem ( X, T ) and a nonempty open set U of X with A ⊃ { n ∈ Z : U ∩ T − n U ∩ . . . ∩ T − dn U = ∅} . As we said before for d = 1 Theorem A can be easily proved. To show Theorem A inthe general case, we need to investigate the properties of F d, . It is interesting thatin the process to do this, generalized polynomials (see § F GP d (resp. F SGP d ) be the family generated by the sets of forms k \ i =1 { n ∈ Z : P i ( n )(mod Z ) ∈ ( − ǫ i , ǫ i ) } , where k ∈ N , P , . . . , P k are generalized polynomials of degree ≤ d (resp. specialgeneralized polynomials), and ǫ i >
0. For the precise definitions see § d Bohr -sets and thesets defined above using generalized polynomials. Theorem B : Let d ∈ N . Then F d, = F GP d . To prove Theorem B we first figure out a subclass of generalized polynomials(called special generalized polynomials) and show that F GP d = F SGP d . When d = 1,we have F , = F SGP . This is the result of Katznelson [23], since F SGP is generatedby sets of forms ∩ ki =1 { n ∈ Z : na i (mod Z ) ∈ ( − ǫ i , ǫ i ) } with k ∈ N , a i ∈ R and ǫ i > Theorem C : Let d ∈ N . If A ∈ F GP d , then there exist a minimal d -step nilsystem ( X, T ) and a nonempty open set U such that A ⊃ { n ∈ Z : U ∩ T − n U ∩ . . . ∩ T − dn U = ∅} . The proof of Theorem B is divided into two parts, namely
Theorem B(1) : F d, ⊂ F GP d and Theorem B(2) : F d, ⊃ F GP d . The proof of Theorem B(1) is a theoretical argument using nilpotent Lie grouptheory; and the proofs of Theorem B(2) and Theorem C are very complicated con-struction and computation where nilpotent matrix Lie group is used.
Remark . Our definition of generalized polynomials is slight different from theones defined in [5]. In fact we need to specialize the degree of the generalized
Nil-Bohr sets polynomials which is not needed in [5]. Moreover, our Theorem B can be comparedwith Theorem A of Bergelson and Leibman proved in [5].In [13, 12] Furstenberg introduced the notion of Poincar´e recurrence sets andBirkhoff recurrence sets. Here is a generalization of the above notion. Let d ∈ N .We say that S ⊂ Z is a set of d -recurrence if for every measure preserving dynamicalsystem ( X, X , µ, T ) and for every A ∈ X with µ ( A ) >
0, there exists n ∈ S suchthat µ ( A ∩ T − n A ∩ . . . ∩ T − dn A ) > . We say that S ⊆ Z is a set of d -topological recurrence if for every minimal system( X, T ) and for every nonempty open subset U of X , there exists n ∈ S such that U ∩ T − n U ∩ . . . ∩ T − dn U = ∅ . Remark . The above definitions are slightly different from the ones introducedin [11], namely we do not require n = 0. The main reason we define in this way isthat for each A ∈ F d, , 0 ∈ A . Thus { } ∪ C ∈ F ∗ d, for each C ⊂ Z .Let F P oi d (resp. F Bir d ) be the family generated by the collection of all sets of d -recurrence (resp. sets of d -topological recurrence). It is obvious by the abovedefinition that F P oi d ⊂ F Bir d . Moreover, it is known that for each d ∈ N , F P oi d % F P oi d +1 and F Bir d % F Bir d +1 [11]. Now we state a problem which is related toProblem B-II. Problem B-III:
Is it true that F Bir d = F ∗ d, ? where F ∗ d, is the dual family of F d, , i.e. the collection of sets intersecting everyNil d Bohr set.An immediate corollary of Theorem A is: Corollary D : Let d ∈ N . Then F P oi d ⊂ F Bir d ⊂ F ∗ d, . Note that F P oi = F Bir [24]. Though we can not prove F Bir d = F ∗ d, , we willshow that the two collections coincide “dynamically”, i.e. both of them can be usedto characterize higher order almost automorphic points, see § Organization of the paper.
We organize the paper as follows: In Section2, we give some basic definitions, and particularly we show the equivalence of theProblems I, II and III. In Section 3 we recall basic facts related to nilpotent Liegroups and nilmanifolds, and study the properties of the metric on nilpotent matrixLie groups. In Section 4, we introduce the notions related to generalized polynomialsand special generalized polynomials, and give the basic properties. In the next threesections we show the main results. And in the last section, we state the applicationsof our main results, which will appear in a forthcoming article [22] by the sameauthors.1.5.
Thanks.
We thank Bergelson, Frantzikinakis and Glasner for useful comments. . Huang, S. Shao and X.D. Ye 5 Preliminaries
In this section we introduce some basic notions related to dynamical systems,explainhow Bergelson-Host-Kra’s result is related to Problem B-II and show the equivalenceof the Problems I, II and III.2.1.
Measurable and topological dynamics.
A (measurable) system is a quadru-ple ( X, X , µ, T ), where ( X, X , µ ) is a Lebesgue probability space and T : X → X isan invertible measure preserving transformation.A topological dynamical system , referred to more succinctly as just a system , is apair ( X, T ), where X is a compact metric space and T : X → X is a homeomor-phism. We use ρ ( · , · ) to denote the metric on X .2.2. Families and filters.
Since many statements of the paper are better statedusing the notion of a family, we now give the definition. See [1] for more details.2.2.1.
Furstenberg families.
We say that a collection F of subsets of Z is a family if it is hereditary upward, i.e. F ⊆ F and F ∈ F imply F ∈ F . A family F iscalled proper if it is neither empty nor the entire power set of Z , or, equivalently if Z ∈ F and ∅ 6∈ F . Any nonempty collection A of subsets of Z generates a family F ( A ) := { F ⊆ Z : F ⊃ A for some A ∈ A} .For a family F its dual is the family F ∗ := { F ⊆ Z : F ∩ F ′ = ∅ for all F ′ ∈ F } .It is not hard to see that F ∗ = { F ⊂ Z : Z \ F
6∈ F } , from which we have that if F is a family then ( F ∗ ) ∗ = F . Filter and Ramsey property.
If a family F is closed under finite intersectionsand is proper, then it is called a filter .A family F has the Ramsey property if A = A ∪ A ∈ F then A ∈ F or A ∈ F .It is well known that a proper family has the Ramsey property if and only if its dual F ∗ is a filter [13].A subset S of Z is syndetic if it has a bounded gap, i.e. there is N ∈ N such that { i, i + 1 , · · · , i + N } ∩ S = ∅ for every i ∈ Z . A subset S is an IP -set , if there is asubsequence { p i } of Z such that S ⊃ { p i + . . . + p i n : i < . . . < i n , n ∈ N } . It is known that the family of all IP ∗ -sets is a filter and each IP ∗ -set is syndetic[13].The upper Banach density and lower Banach density of S are BD ∗ ( S ) = lim sup | I |→∞ | S ∩ I || I | , and BD ∗ ( S ) = lim inf | I |→∞ | S ∩ I || I | , where I ranges over intervals of Z , while the upper density of S and the lower density of S are D ∗ ( S ) = lim sup n →∞ | S ∩ [ − n, n ] | n + 1 , and D ∗ ( S ) = lim inf n →∞ | S ∩ [ − n, n ] | n + 1 . If D ∗ ( S ) = D ∗ ( S ), then we say the density of S is D ( S ) = D ∗ ( S ) = D ∗ ( S ). Nil-Bohr sets A Bergelson-Host-Kra’ Theorem and a consequence.
In this subsectionwe explain how Bergelson-Host-Kra’s result is related to Problem B-II. First we needsome definitions.
Definition 2.1.
Let k ≥ X = G/ Γ be a d -step nilmanifold.Let φ be a continuous real (or complex) valued function on X and let a ∈ G and b ∈ X . The sequence { φ ( a n · b ) } is called a basic d -step nilsequence. A d -stepnilsequence is a uniform limit of basic d -step nilsequences.For the definition of nilmanifolds see Section 3. Definition 2.2.
Let { a n : n ∈ Z } be a bounded sequence. We say that a n tends tozero in uniform density, and we write UD-Lim a n = 0 , iflim N −→ + ∞ sup M ∈ Z M + N − X n = M | a n | = 0 . Equivalently, UD-Lim a n = 0 if and only if for any ǫ > , the set { n ∈ Z : | a n | > ǫ } has upper Banach density zero. Now we state their result. Theorem 2.3 (Bergelson-Host-Kra) . [4, Theorem 1.9] Let ( X, X , µ, T ) be an ergodicsystem, let f ∈ L ∞ ( µ ) and let d ≥ be an integer. The sequence { I f ( d, n ) } is thesum of a sequence tending to zero in uniform density and a d -step nilsequence, where (2.1) I f ( d, n ) = Z f ( x ) f ( T n x ) . . . f ( T dn x ) dµ ( x ) . Especially, for any A ∈ X (2.2) { I A ( d, n ) } = { µ ( A ∩ T − n A ∩ . . . ∩ T − dn A ) } = F d + N, where F d is a d -step nilsequence and N tending to zero in uniform density. Regard F d as a function F d : Z → C . By [20] there is a d -step nilsystem ( Z, S ), x ∈ Z anda continuous function φ ∈ C ( Z ) such that F d ( n ) = φ ( S n x ) . We claim that φ ( x ) > µ ( A ) >
0. Assume that contrary that φ ( x ) ≤
0. By[14] or [6, Theorem 6.15] there is c > { n ∈ Z : µ ( A ∩ T − n A ∩ . . . ∩ T − dn A ) > c } is an IP ∗ -set. On the other hand there is a small neighborhood V of x such that φ ( x ) < c for each x ∈ V by the continuity of φ . It is known that N ( x , V ) is an IP ∗ -set [13] since ( Z, S ) is distal [2, Ch 4, Theorem 3] or [25]. This contradicts to(2.2) by the facts that the family of IP ∗ -sets is a filter, each IP ∗ -set is syndetic and N ( n ) tends to zero in uniform density. That is, we have shown that φ ( x ) > µ ( A ) > µ ( A ) > ǫ > { n ∈ Z : φ ( S n x ) > φ ( x ) − ǫ } is aNil d -Bohr set. Since { n ∈ Z : | N ( n ) | > ǫ } has zero upper Banach density we havethe following corollary . Huang, S. Shao and X.D. Ye 7 Corollary 2.4.
Let ( X, X , µ, T ) be an ergodic system and d ∈ N . Then for all A ∈ X with µ ( A ) > and ǫ > , the set I = { n ∈ Z : µ ( A ∩ T − n A ∩ . . . ∩ T − dn A ) > φ ( x ) − ǫ } is an almost Nil d Bohr -set, i.e. there is some subset M with BD ∗ ( M ) = 0 suchthat I ∆ M is a Nil d Bohr -set. It follows that problem B-II(2) has a positive answer ignoring a set with zerodensity, since for a minimal system (
X, T ), each invariant measure of (
X, T ) is fullysupported.2.4.
Furstenberg correspondence principle.
Let F ( Z ) denote the collection offinite non-empty subsets of Z . It is well known that Theorem 2.5 (Topological case) . (1) Let E ⊆ Z be a syndetic set. Then thereexist a minimal system ( X, T ) and a non-empty open set U ⊆ X such that { α ∈ F ( Z ) : \ n ∈ α T − n U = ∅} ⊆ { α ∈ F ( Z ) : \ n ∈ α ( E − n ) = ∅} . (2) For any minimal system ( X, T ) and any open non-empty set U , there is asyndetic set E such that { α ∈ F ( Z ) : \ n ∈ α ( E − n ) = ∅} ⊆ { α ∈ F ( Z ) : \ n ∈ α T − n U = ∅} . Theorem 2.6 (Measurable case) . (1) Let E ⊆ Z with BD ∗ ( E ) > . Then thereexists a measurable system ( X, X , µ, T ) and A ∈ X with µ ( A ) = BD ∗ ( E ) such that for all α ∈ F ( Z ) BD ∗ ( \ n ∈ α ( E − n )) ≥ µ ( \ n ∈ α T − n A ) . (2) Let ( X, X , µ, T ) be a measurable system and A ∈ X with µ ( A ) > . There isa set E with D ∗ ( E ) ≥ µ ( A ) such that { α ∈ F ( Z ) : \ n ∈ α ( E − n ) = ∅} ⊆ { α ∈ F ( Z ) : µ ( \ n ∈ α T − n U ) > } . Equivalence.
In this subsection we explain why Problems B-I,II,III are equiv-alent. Let F be the family generated by all sets of forms { n ∈ Z : U ∩ T − n U ∩ . . . ∩ T − dn U = ∅} , with ( X, T ) a minimal system, U a non-empty open subset of X . Thenit is clear from the definition that F Bir d = F ∗ . Proposition 2.7.
For any d ∈ N the following statements are equivalent. (1) For any Nil d Bohr -set A , there is a syndetic subset S of Z with A ⊃ { n ∈ Z : S ∩ ( S − n ) ∩ . . . ∩ ( S − dn ) = ∅} . (2) For any Nil d Bohr -set A , there are a minimal system ( X, T ) and a non-empty open subset U of X with A ⊃ { n ∈ Z : U ∩ T − n U ∩ . . . ∩ T − dn U = ∅} . (3) F Bir d ⊂ F ∗ d, . Nil-Bohr sets Proof.
Let d ∈ N be fixed. (1) ⇒ (2). Let A be a Nil d Bohr -set, then there is asyndetic subset S of Z with A ⊃ { n ∈ Z : S ∩ ( S − n ) ∩ . . . ∩ ( S − dn ) = ∅} . For such S using Theorem 2.5, we get that there exist a minimal system ( X, T ) and a non-empty open set U ⊆ X such that { n ∈ Z : S ∩ ( S − n ) ∩ . . . ∩ ( S − dn ) = ∅} ⊃ { n ∈ Z : U ∩ T − n U ∩ . . . ∩ T − dn U = ∅} . Thus A ⊃ { n ∈ Z : U ∩ T − n U ∩ . . . ∩ T − dn U = ∅} . (2) ⇒ (1) follows similarly by the above argument. (2) ⇒ (3) follows by the definition.(3) ⇒ (2). Since F Bir d ⊂ F ∗ d, and F Bir d = F ∗ , we have that F ∗ ⊂ F ∗ d, which impliesthat F ⊃ F d, . (cid:3) Proposition 2.8.
For any d ∈ N the following statements are equivalent. (1) For any syndetic set S , { n ∈ Z : S ∩ ( S − n ) ∩ . . . ∩ ( S − dn ) = ∅} is a Nil d Bohr -set. (2) For any minimal system ( X, T ) , and any open non-empty U ⊂ X , { n ∈ Z : U ∩ T − n U ∩ . . . ∩ T − dn U = ∅} is a Nil d Bohr -set. (3) F Bir d ⊃ F ∗ d, .Proof. Let d ∈ N be fixed. (1) ⇒ (2). Let ( X, T ) be a minimal system and U be anon-empty open set of X . By Theorem 2.5, there is a syndetic set S such that { n ∈ Z : S ∩ ( S − n ) ∩ . . . ∩ ( S − dn ) = ∅} ⊂ { n ∈ Z : U ∩ T − n U ∩ . . . ∩ T − dn U = ∅} . By (1), { n ∈ Z : S ∩ ( S − n ) ∩ . . . ∩ ( S − dn ) = ∅} is a Nil d Bohr -set, and so is { n ∈ Z : U ∩ T − n U ∩ . . . ∩ T − dn U = ∅} . Similarly, we have (2) ⇒ (1). (2) ⇒ (3) followsby the definition. (3) ⇒ (2). By Corollary D we have F ∗ d, = F ∗ , i.e. F d, = F . (cid:3) Other related problems.
We remark that similar problems can be formu-lated replacing syndetic sets by sets with positive Banach density, minimal systemsby ergodic systems and open non-empty sets by positive measurable sets.3.
Nilsystems
In this section we recall some basic facts concerning nilpotent Lie groups andnilmanifolds. Since in the proofs of our main results we need to use the metric inthe nilpotent matrix Lie group, we state some its basic properties. Note that wefollow Green and Tao [15] to define such a metric.3.1.
Nilmanifolds and nilsystems.
Nilpotent groups.
Let G be a group. For g, h ∈ G , we write [ g, h ] = ghg − h − for the commutator of g and h and we write [ A, B ] for the subgroup spanned by { [ a, b ] : a ∈ A, b ∈ B } . The commutator subgroups G j , j ≥
1, are defined inductivelyby setting G = G and G j +1 = [ G j , G ]. Let d ≥ G is d -step nilpotent if G d +1 is the trivial subgroup.3.1.2. Nilmanifolds.
Let G be a d -step nilpotent Lie group and Γ a discrete cocom-pact subgroup of G , i.e. a uniform subgroup of G . The compact manifold X = G/ Γis called a d -step nilmanifold . The group G acts on X by left translations and wewrite this action as ( g, x ) gx . The Haar measure µ of X is the unique probabilitymeasure on X invariant under this action. Let τ ∈ G and T be the transformation . Huang, S. Shao and X.D. Ye 9 x τ x of X , i.e the nilrotation induced by τ ∈ G . Then ( X, T, µ ) is called a basic d -step nilsystem . See [10, 26] for the details.3.1.3. d -step nilsystem and system of order d . We also make use of inverse limits ofnilsystems and so we recall the definition of an inverse limit of systems (restrictingourselves to the case of sequential inverse limits). If ( X i , T i ) i ∈ N are systems with diam ( X i ) ≤ M < ∞ and φ i : X i +1 → X i are factor maps, the inverse limit of thesystems is defined to be the compact subset of Q i ∈ N X i given by { ( x i ) i ∈ N : φ i ( x i +1 ) = x i , i ∈ N } , which is denoted by lim ←− { X i } i ∈ N . It is a compact metric space endowedwith the distance ρ ( x, y ) = P i ∈ N / i ρ i ( x i , y i ). We note that the maps { T i } inducea transformation T on the inverse limit. Definition 3.1. [Host-Kra-Maass] [19] A system (
X, T ) is called a d -step nilsystem ,if it is an inverse limit of basic d -step nilsystems. A system ( X, T ) is called a systemof order d , if it is a minimal d -step nilsystem, equivalently it is an inverse limit ofbasic d -step minimal nilsystems.Recall that a subset A ⊆ Z is a Nil d Bohr -set if there exist a d -step nilsystem( X, T ), x ∈ X and an open set U ⊆ X containing x such that N ( x , U ) is containedin A . As each basic d -step nilsystem is distal, so is a d -step nilsystem. Hence byDefinition 3.1, it is not hard to see that a subset A ⊆ Z is a Nil d Bohr -set if andonly if there exist a basic d -step (minimal) nilsystem ( X, T ) (or a system ( X, T ) of order d ), x ∈ X and an open set U ⊆ X containing x such that N ( x , U ) iscontained in A . Note that here we need the facts that the product of finitely manyof d -step nilmanifolds is a d -step nilmanifold, and the orbit closure of any point ina basic d -step nilsystem is a d -step nilmanifold [25, Theorem 2.21].3.2. Reduction.
Let X = G/ Γ be a nilmanifold. Then there exists a connected,simply connected nilpotent Lie group b G and b Γ ⊆ b G a co-compact subgroup such that X with the action of G is isomorphic to a submanifold e X of b X = b G/ b Γ representingthe action of G in b G . See [25] for more details.Thus a subset A ⊆ Z is a Nil d Bohr -set if and only if there exist a basic d -stepnilsystem ( G/ Γ , T ) with G is a connected, simply connected nilpotent Lie group andΓ a co-compact subgroup of G , x ∈ X and an open set U ⊆ X containing x suchthat N ( x , U ) is contained in A .3.3. Nilpotent Lie group and Mal’cev basis. g of a d -step nilpotent Lie group G together with the exponential map exp : g −→ G . When G is a connected, simply-connected d -step nilpotent Lie group the exponential map is a diffeomorphism [10,26]. In particular, we have a logarithm map log : G −→ g . Letexp( X ∗ Y ) = exp( X )exp( Y ) , X, Y ∈ g . sets Campbell-Baker-Hausdorff formula.
The following Campbell-Baker-Hausdorffformula (CBH formula) will be used frequently X ∗ Y = X n> ( − n +1 n X p i + q i > , ≤ i ≤ n ( P ni =1 ( p i + q i )) − p ! q ! . . . p n ! q n ! × (ad X ) p (ad Y ) q . . . (ad X ) p n (ad Y ) q n − Y, where (ad X ) Y = [ X, Y ]. (If q n = 0, the term in the sum is . . . (ad X ) p n − X ; ofcourse if q n >
1, or if q n = 0 and p n >
1, then the term if zero.) The low ordernonzero terms are well known, X ∗ Y = X + Y + 12 [ X, Y ] + 112 [ X, [ X, Y ]] −
112 [ Y, [ X, Y ]] −
148 [ Y, [ X, [ X, Y ]]] −
148 [ X, [ Y, [ X, Y ]]]+ ( commutators in five or more terms) . g is the Lie algebra of G over R , and exp : g −→ G is theexponential map. The descending central series of g is defined inductively by g (1) = g ; g ( n +1) = [ g , g ( n ) ] = span { [ X, Y ] : X ∈ g , Y ∈ g ( n ) } . Since g is a d -step nilpotent Lie algebra, we have g = g (1) ) g (2) ) . . . ) g ( d ) ) g ( d +1) = { } . We note that [ g ( i ) , g ( j ) ] ⊂ g ( i + j ) , ∀ i, j ∈ N . In particular, each g ( k ) is an ideal in g .3.3.4. Mal’cev Base.
Definition 3.2. (Mal’cev base) Let G/ Γ be an m -dimensional nilmanifold (i.e. G is a d -step nilpotent Lie group and Γ is a discrete uniform subgroup of G ) and let G = G ⊃ . . . ⊃ G d ⊃ G d +1 = { e } be the lower central series filtration. A basis X = { X , . . . , X m } for the Lie algebra g over R is called a Mal’cev basis for G/ Γ ifthe following four conditions are satisfied:(1) For each j = 0 , . . . , m − η j := Span( X j +1 , . . . , X m ) is a Liealgebra ideal in g , and hence H j := exp η j is a normal Lie subgroup of G .(2) For every 0 < i < d we have G i = H l i − +1 . Thus 0 = l < l < . . . < l d − ≤ m − g ∈ G can be written uniquely as exp( t X )exp( t X ) . . . exp( t m X m ),for t i ∈ R .(4) Γ consists precisely of those elements which, when written in the above form,have all t i ∈ Z .Note that such a basis exists when G is a connected, simply connected d -stepnilpotent Lie group [10, 15, 26]. . Huang, S. Shao and X.D. Ye 11 Metrics on nilmanifolds.
For a connected, simply connected d -step nilpotentLie group G , we can use a Mal’cev basis X to put a metric structure on G and on G/ Γ. Definition 3.3 (Metrics on G and G/ Γ) . [15] Let G/ Γ be a nilmanifold with aMal’cev basis X , where G is a connected, simply connected Lie group and Γ is adiscrete uniform subgroup of G . Let φ : G −→ R m with g = exp( t X ) . . . exp( t m X m ) ( t , . . . , t m ) . We define ρ = ρ X : G × G −→ R to be the largest metric such that ρ ( x, y ) ≤| φ ( xy − ) | for all x, y ∈ G , where | · | denotes the ℓ ∞ -norm on R m . More explicitly,we have ρ ( x, y ) = inf n n X i =1 min {| φ ( x i − x − i ) | , | φ ( x i x − i − ) |} : x , . . . , x n ∈ G ; x = x, x n = y o . This descends to a metric on G/ Γ by setting ρ ( x Γ , y Γ) := inf { d ( x ′ , y ′ ) : x ′ , y ′ ∈ G ; x ′ = x (mod Γ); y ′ = y (mod Γ) } . It turns out that this is indeed a metric on G/ Γ, see [15]. Since ρ is right-invariant,we also have ρ ( x Γ , y Γ) = inf γ ∈ Γ ρ ( x, yγ ) . Base points.
The following proposition should be well known.
Proposition 3.4.
Let X = G/ Γ be a nilmanifold, T be a nilrotation induced by a ∈ G . Let x ∈ G and U be an open neighborhood x Γ in X . Then there are auniform subgroup Γ x ⊂ G and an open neighborhood V ⊂ G/ Γ x of e Γ x such that N T ( x Γ , U ) = N T ′ ( e Γ x , V ) , where T ′ is a nilrotation induced by a ∈ G in X ′ = G/ Γ x .Proof. Let Γ x = x Γ x − . Then Γ x is also a uniform subgroup of G .Put V = U x − , where we view U as the collections of equivalence classes. It iseasy to see that V ⊂ G/ Γ x is open, which contains e Γ x . Let n ∈ N T ( x Γ , U ) then a n x Γ ∈ U which implies that a n x Γ x − ∈ U x − = V , i.e. n ∈ N T ′ ( e Γ x , V ). Theother direction follows similarly. (cid:3) Nilpotent Matrix Lie Group. M d +1 ( R ) denote the space of all ( d + 1) × ( d + 1)-matrices with realentries. For A = ( A ij ) ≤ i,j ≤ d +1 ∈ M d +1 ( R ), we define(3.1) k A k = d +1 X i,j =1 | A ij | ! . Then k · k is a norm on M d +1 ( R ) and the norm satisfies the inequalities k A + B k ≤ k A k + k B k and k AB k ≤ k A kk B k for A, B ∈ M d +1 ( R ). sets a = ( a ki ) ≤ k ≤ d, ≤ i ≤ d − k +1 ∈ R d ( d +1) / . Then corresponding to a we define M ( a ) with M ( a ) = a a a . . . a d − a d a a . . . a d − a d − a . . . a d − a d − ... ... ... ... ... ... ...0 0 0 0 . . . a d − a d − . . . a d . . . . G d be the (full) upper triangular group G d = { M ( a ) : a ki ∈ R , ≤ k ≤ d, ≤ i ≤ d − k + 1 } . The group G d is a d -step nilpotent group, and it is clear that for A ∈ G d there existsa unique c = ( c ki ) ≤ k ≤ d, ≤ i ≤ d − k +1 ∈ R d ( d +1) / such that A = M ( c ). LetΓ = { M ( h ) : h ki ∈ Z , ≤ k ≤ d, ≤ i ≤ d − k + 1 } . Then Γ is a uniform subgroup of G d .3.6.4. Let a = ( a ki ) ≤ k ≤ d, ≤ i ≤ d − k +1 ∈ R d ( d +1) / and b = ( b ki ) ≤ k ≤ d, ≤ i ≤ d − k +1 ∈ R d ( d +1) / . If c = ( c ki ) ≤ k ≤ d, ≤ i ≤ d − k +1 ∈ R d ( d +1) / such that M ( c ) = M ( a ) M ( b ),then(3.2) c ki = k X j =0 a k − ji b ji + k − j = a ki + ( k − X j =1 a k − ji b ji + k − j ) + b ki for 1 ≤ k ≤ d and 1 ≤ i ≤ d − k + 1, where we assume a = a = . . . = a d = 1 and b = b = . . . = b d = 1.3.6.5. Now we endow a compatible metric ρ on G d and G d / Γ. Definition 3.5 (Metric on G d ) . Let I be the ( d + 1) × ( d + 1) identity matrix.Define ρ : G d × G d −→ R such that ρ ( A, B ) = inf { n X i =1 min {k A i − A − i − I k , k A i A − i − − I k} : A , . . . , A n ∈ G ; A = A, A n = B } . Lemma 3.6.
For any
A, B ∈ G d , ρ ( A, B ) ≤ k AB − − I k ≤ k A − B kk B − k and k A − B kk B k ≤ k AB − − I k ≤ e ρ ( A,B )+ ρ ( A,B ) + ··· + ρ ( A,B ) d − . Proof.
It is clear that it is sufficient to prove(3.3) k AB − − I k ≤ e ρ ( A,B )+ ρ ( A,B ) + ··· + ρ ( A,B ) d − . Huang, S. Shao and X.D. Ye 13 for A, B ∈ G d . The others are obvious. Let A, B ∈ G d . For any ǫ >
0, there exist A , . . . , A n ∈ G d ; A = A, A n = B such that n X i =1 min {k A i − A − i − I k , k A i A − i − − I k} ≤ ρ ( A, B ) + ǫ. Let t i = min {k A i − A − i − I k , k A i A − i − − I k} for i = 1 , , · · · , n . Note that for C ∈ G d , C − = I + P di =1 ( I − C ) i . Hence k C − − I k ≤ d X i =1 k C − I k i . Moreover, if we set t = min {k C − I k , k C − − I k} , then(3.4) k C − − I k ≤ t (1 + t + t + · · · + t d − ) . Then by (3.4), k A i − A − i − I k ≤ t i (1 + t i + t i + · · · + t d − i ) ≤ t i (1 + ( ρ ( A, B ) + ǫ ) + · · · + ( ρ ( A, B ) + ǫ ) d − )for i = 1 , , · · · , n . Thus n X i =1 k A i − A − i − I k ≤ n X i =1 t i (1 + ( ρ ( A, B ) + ǫ ) + · · · + ( ρ ( A, B ) + ǫ ) d − ) ≤ ( ρ ( A, B ) + ǫ ) + · · · + ( ρ ( A, B ) + ǫ ) d . Now1 + k AB − − I k = 1 + k A A − n − I k = 1 + k A A − A A − n − I k = 1 + k ( A A − − I )( A A − n − I ) + ( A A − − I ) + ( A A − n − I ) k≤ k ( A A − − I ) kk ( A A − n − I ) k + k ( A A − − I ) k + k ( A A − n − I ) k = (1 + k A A − − I k )(1 + k A A − n − I k ) . Continuing this process we get that1 + k AB − − I k ≤ (1 + k A A − − I k )(1 + k A A − − I k ) · · · (1 + k A n − A − n − I k ) ≤ e k A A − − I k + k A A − − I k + ··· + k A n − A − n − I k = e ( ρ ( A,B )+ ǫ )+ ··· +( ρ ( A,B )+ ǫ ) d This implies k AB − − I k ≤ e ( ρ ( A,B )+ ǫ )+ ··· +( ρ ( A,B )+ ǫ ) d −
1. Let ǫ ց
0, we get (3.3).This ends the proof of the lemma. (cid:3)
Proposition 3.7 (Metrics on G d and G d / Γ) . ρ is a right-invariant metric on G d .This descends to a metric on G d / Γ by setting ρ ( A Γ , B Γ) := inf { ρ ( Aγ, Bγ ′ ) : γ, γ ′ ∈ Γ } . Since ρ is right-invariant, we also have ρ ( A Γ , B Γ) = inf γ ∈ Γ ρ ( A, Bγ ) . sets Proof.
Firstly, it is clear that ρ : G d × G d −→ R is a right-invariant, non-negativefunction and for A, B, C ∈ G d , ρ ( A, B ) = ρ ( B, A ) and ρ ( A, C ) ≤ ρ ( A, B ) + ρ ( B, C ) . By Lemma 3.6 ρ ( A, B ) = 0 if and only if k A − B k = 0, i.e., A = B . Thus ρ is aright-invariant metric on G d . Moreover by Lemma 3.6, we know that the metric ρ is equivalent to the metric induced by the norm k · k on G d . Thus, ρ is a compatiblemetric with topology of G d .Next we are going to show that this descends to a metric on G d / Γ by setting ρ ( A Γ , B Γ) := inf { ρ ( Aγ, Bγ ′ ) : γ, γ ′ ∈ Γ } . Since ρ is a right-invariant metric on G d , it is sufficient to show that if ρ ( A Γ , B Γ) = 0then A Γ = B Γ. Suppose ρ ( A Γ , B Γ) = 0. Since ρ is right-invariant, inf γ ∈ Γ ρ ( Aγ, B ) =0. Moreover we can find γ i ∈ Γ such that k B k ( e ρ ( Aγ i ,B )+ ··· + ρ ( Aγ i ,B ) d − < i (1+ k A − k ) for each i ∈ N . By Lemma 3.6, we have k Aγ i − B k ≤ k B k ( e ρ ( Aγ i ,B )+ ··· + ρ ( Aγ i ,B ) d − < i (1 + k A − k )for i ∈ N . Thus for all i ≤ j ∈ N , k γ i − γ j k = k A − ( A ( γ i − B ) − A ( γ j − B )) k≤ k A − k ( k Aγ i − B k + k Aγ j − B k ) < k A − k ( 12 i (1 + k A − k ) + 12 i (1 + k A − k ) ) < . Since γ i , γ j ∈ Γ, this implies γ i = γ j for i, j ∈ N . Thus k Aγ − B k = k Aγ j − B k < j (1 + k A − k )for any j ∈ N . Hence k Aγ − B k = 0. So Aγ = B and A Γ = B Γ. This ends theproof of the proposition. (cid:3) Generalized polynomials
In this section we introduce the notions and basic properties of (special) general-ized polynomials. It will be used in the following sections.4.1.
Definitions. a ∈ R , let || a || = inf {| a − n | : n ∈ Z } and ⌈ a ⌉ = min { m ∈ Z : | a − m | = || a ||} . When studying F d, we find that the generalized polynomials appear naturally.Here is the precise definition. Note that we use f ( n ) or f to denote the generalizedpolynomials. . Huang, S. Shao and X.D. Ye 15 Generalized polynomials.
Definition 4.1.
Let d ∈ N . We define the generalized polynomials of degree ≤ d (denoted by GP d ) by induction. For d = 1, GP is the collection of functions from Z to R containing h a , a ∈ R with h a ( n ) = an for each n ∈ Z which is closed undertaking ⌈ ⌉ , multiplying by a constant and the finite sums.Assume that GP i is defined for i < d . Then GP d is the collection of functionsfrom Z to R containing GP i with i < d , functions of the forms a n p ⌈ f ( n ) ⌉ . . . ⌈ f k ( n ) ⌉ (with a ∈ R , p ≥ k ≥ f l ∈ GP p l and P kl =0 p l = d ), which is closed undertaking ⌈ ⌉ , multiplying by a constant and the finite sums. Let GP= ∪ ∞ i =1 GP i .For example, a ⌈ a ⌈ a n ⌉⌉ + b n ∈ GP , and a ⌈ a n ⌉ + b ⌈ b ⌈ b n ⌉⌉ + c n + c n ∈ GP , where a i , b i , c i ∈ R . Note that if f ∈ GP then f (0) = 0.4.1.3. Special generalized polynomials.
Since generalized polynomials are very com-plicated, we will specify a subclass of them, called the special generalized polynomials which will be used in our proofs of the main results. To do this, we need some no-tions.For a ∈ R , we define L ( a ) = a . For a , a ∈ R we define L ( a , a ) = a ⌈ L ( a ) ⌉ .Inductively, for a , a , · · · , a ℓ ∈ R ( ℓ ≥
2) we define(4.1) L ( a , a , · · · , a ℓ ) = a ⌈ L ( a , a , · · · , a ℓ ) ⌉ . For example, L ( a , a , a ) = a ⌈ a ⌈ a ⌉⌉ .We give now the precise definition of special generalized polynomials. Definition 4.2.
For d ∈ N we define special generalized polynomials of degree ≤ d ,denoted by SGP d as follows. SGP d is the collection of generalized polynomialsof the forms L ( n j a , · · · , n j ℓ a ℓ ), where 1 ≤ ℓ ≤ d, a , · · · , a ℓ ∈ R , j , · · · , j ℓ ∈ N with P ℓt =1 j t ≤ d. Thus SGP = { an : a ∈ R } , SGP = { an , bn ⌈ cn ⌉ , en : a, b, c, e ∈ R } andSGP =SGP ∪ { an , an ⌈ bn ⌉ , an ⌈ bn ⌉ , an ⌈ bn ⌈ cn ⌉⌉ : a, b, c ∈ R } .4.1.4. F GP d and F SGP d . Let F GP d be the family generated by the sets of forms k \ i =1 { n ∈ Z : P i ( n ) (mod Z ) ∈ ( − ǫ i , ǫ i ) } , where k ∈ N , P i ∈ GP d , and ǫ i >
0, 1 ≤ i ≤ k . Note that P i ( n ) (mod Z ) ∈ ( − ǫ i , ǫ i )if and only if || P i ( n ) || < ǫ i . Let F SGP d be the family generated by the sets of forms k \ i =1 { n ∈ Z : P i ( n ) (mod Z ) ∈ ( − ǫ i , ǫ i ) } , where k ∈ N , P i ∈ SGP d , and ǫ i >
0, 1 ≤ i ≤ k . Note that from the definition both F GP d and F SGP d are filters; and F SGP d ⊂ F GP d . sets Basic properties of generalized polynomials. f ∈ GP we let f ∗ = −⌈ f ⌉ . Lemma 4.3.
Let c ∈ R and f , . . . , f k ∈ GP with k ∈ N . Then c ⌈ f ⌉ . . . ⌈ f k ⌉ = c ( − k k Y i =1 ( f i − ⌈ f i ⌉ ) − c ( − k X i ,...,ik ∈{ , ∗} ( i ,...,ik ) =( ∗ ,..., ∗ ) f i . . . f i k k . Particularly if k = 2 we get that c ⌈ f ⌉⌈ f ⌉ = cf ⌈ f ⌉ − cf f + cf ⌈ f ⌉ + c ( f − ⌈ f ⌉ )( f − ⌈ f ⌉ ) . Proof.
Expanding Q ki =1 ( f i − ⌈ f i ⌉ ) we get that k Y i =1 ( f i − ⌈ f i ⌉ ) = X i ,...,i k ∈{ , ∗} f i . . . f i k k . So we have c ⌈ f ⌉ . . . ⌈ f k ⌉ = c ( − k k Y i =1 ( f i − ⌈ f i ⌉ ) − c ( − k X i ,...,ik ∈{ , ∗} ( i ,...,ik ) =( ∗ ,..., ∗ ) f i . . . f i k k . (cid:3) Let c = 1 in Lemma 4.3 we have Lemma 4.4.
Let f , f , . . . , f k ∈ GP . Then f ⌈ f ⌉ . . . ⌈ f k ⌉ = ( − k − k Y i =1 ( f i − ⌈ f i ⌉ ) + ( − k X i ,...,ik ∈{ , ∗} ( i ,...,ik ) =(1 , ∗ ,..., ∗ ) f i . . . f i k k . Particularly if k = 2 we have f ⌈ f ⌉ = ⌈ f ⌉⌈ f ⌉ + f f − f ⌈ f ⌉ − ( f − ⌈ f ⌉ )( f − ⌈ f ⌉ ) . Let k = 1 in Lemma 4.3 we have Lemma 4.5.
Let c ∈ R and f ∈ GP . Then c ⌈ f ⌉ = cf − c ( f − ⌈ f ⌉ ) . F GP d = F SGP d . To do this we useinduction. To make the proof clearer, first we give some results under the assumption(4.2) F GP d − ⊂ F SGP d − . Definition 4.6.
Let r ∈ N with r ≥
2. We define SW r = { ℓ Y i =1 ( w i ( n ) − ⌈ w i ( n ) ⌉ ) : ℓ ≥ , r i ≥ , w i ( n ) ∈ GP r i and ℓ X i =1 r i ≤ r } and W r = R − Span {SW r } , . Huang, S. Shao and X.D. Ye 17 that is, W r = { ℓ X j =1 a j p j ( n ) : ℓ ≥ , a j ∈ R , p j ( n ) ∈ SW r for each j = 1 , , · · · , ℓ } . Lemma 4.7.
Under the assumption (4.2) , one has for any p ( n ) ∈ W d and ǫ > , { n ∈ Z : p ( n ) ( mod Z ) ∈ ( − ǫ, ǫ ) } ∈ F SGP d − . Proof.
Since F SGP d is a filter, it is sufficient to show that for any p ( n ) = aq ( n ) and > δ > q ( n ) ∈ SW d and a ∈ R , { n ∈ Z : p ( n ) (mod Z ) ∈ ( − δ, δ ) } ∈ F SGP d − . Note that as q ( n ) ∈ SW d , there exist ℓ ≥ , r i ≥ , w i ( n ) ∈ GP r i and ℓ P i =1 r i ≤ d suchthat q ( n ) = ℓ Q i =1 ( w i ( n ) − ⌈ w i ( n ) ⌉ ). Since ℓ ≥
2, one has r ≤ d − w ( n ) ∈ GP d − . By the assumption (4.2), { n ∈ Z : w ( n ) (mod Z ) ∈ ( − δ | a | , δ | a | ) } ∈F SGP d − . By the inequality | q ( n ) | ≤ | a || w ( n ) − ⌈ w ( n ) ⌉| for n ∈ Z , we get that { n ∈ Z : p ( n ) (mod Z ) ∈ ( − δ, δ ) } ⊃ { n ∈ Z : | w ( n ) − ⌈ w ( n ) ⌉| ∈ ( − δ | a | , δ | a | ) } = { n ∈ Z : w ( n ) (mod Z ) ∈ ( − δ | a | , δ | a | ) } . Thus { n ∈ Z : p ( n ) (mod Z ) ∈ ( − δ, δ ) } ∈ F SGP d − since { n ∈ Z : w ( n ) (mod Z ) ∈ ( − δ | a | , δ | a | ) } ∈ F SGP d − . (cid:3) Definition 4.8.
Let r ∈ N with r ≥
2. For q ( n ) , q ( n ) ∈ GP r we define q ( n ) ≃ r q ( n )if there exist h ( n ) ∈ GP r − and h ( n ) ∈ W r such that q ( n ) = q ( n ) + h ( n ) + h ( n ) (mod Z )for all n ∈ Z . Lemma 4.9.
Let p ( n ) ∈ GP r and q ( n ) ∈ GP t , r, t ∈ N . Then (1) p ( n ) ⌈ q ( n ) ⌉ ≃ r + t ( p ( n ) − ⌈ p ( n ) ⌉ ) q ( n ) . (2) if q ( n ) , q ( n ) , · · · , q k ( n ) ∈ GP t such that q ( n ) = P ki =1 q i ( n ) , then p ( n ) ⌈ q ( n ) ⌉ ≃ r + t k X i =1 p ( n ) ⌈ q i ( n ) ⌉ . Proof. (1) follows from Lemma 4.4 and (2) follows from (1). (cid:3)
Definition 4.10.
For r ∈ N , we define GP ′ r = { p ∈ GP r : { n ∈ Z : p ( n ) (mod Z ) ∈ ( − ǫ, ǫ ) } ∈ F SGP r for any ǫ > } . Proposition 4.11.
It is clear that sets (1) For p ( n ) ∈ GP r , p ( n ) ∈ GP ′ r if and only if − p ( n ) ∈ GP ′ r . (2) If p ( n ) , p ( n ) , · · · , p k ( n ) ∈ GP ′ r then p ( n ) = p ( n ) + p ( n ) + · · · + p k ( n ) ∈ GP ′ r . (3) F GP d ⊂ F SGP d if and only if GP ′ d = GP d .Proof. (1) can be verified directly. (2) follows from the fact that for each ǫ > { n ∈ Z : p ( n ) (mod Z ) ∈ ( − ǫ, ǫ ) } ⊃ ∩ ki =1 { n ∈ Z : p i ( n ) (mod Z ) ∈ ( − ǫ/k, ǫ/k ) } . (3)follows from the definition of GP ′ d . (cid:3) Lemma 4.12.
Let p ( n ) , q ( n ) ∈ GP d with p ( n ) ≃ d q ( n ) . Under the assumption (4.2) , p ( n ) ∈ GP ′ d if and only if q ( n ) ∈ GP ′ d .Proof. It follows from Lemma 4.7 and the fact that F SGP d is a filter. (cid:3) F GP d = F SGP d .Theorem 4.13. F GP d = F SGP d . Proof.
It is easy to see that F SGP d ⊂ F GP d . So it remains to show F GP d ⊂ F SGP d . That is, if A ∈ F GP d then there is A ′ ∈ F SGP d with A ⊃ A ′ . We will use inductionto show the proposition.Assume first d = 1. In this case we let GP (0) = { g a : a ∈ R } , where g a ( n ) = an for each n ∈ Z . Inductively if GP (0) , . . . , GP ( k ) have been defined then f ∈ GP ( k + 1) if and only if f ∈ GP \ ( S kj =0 GP ( j )) and there are k + 1 ⌈ ⌉ in f . It isclear that GP = ∪ ∞ k =1 GP ( k ). If f ∈ GP (0) then it is clear that f ∈ GP ′ . Assumethat GP (0) , . . . , GP ( k ) ⊂ GP ′ for some k ∈ Z + .Let f ∈ GP ( k + 1). We are going to show that f ∈ GP ′ . If f = f + f with f , f ∈ S ki =0 GP ( i ), then by the above assumption and Proposition 4.11 weconclude that f ∈ GP ′ . The remaining case is f = c ⌈ f ⌉ + f with c ∈ R \ { } , f ∈ GP ( k ), and f ∈ GP (0). By Proposition 4.11 and the fact GP (0) ⊆ GP ′ , f ∈ GP ′ if and only if c ⌈ f ⌉ ∈ GP ′ . So it remains to show c ⌈ f ⌉ ∈ GP ′ . By Lemma4.5 we have c ⌈ f ⌉ = cf − c ( f − ⌈ f ⌉ ). It is clear that cf ∈ GP ( k ) ⊂ GP ′ since f ∈ GP ( k ) ⊂ GP ′ . For any ǫ > { n ∈ Z : || − c ( f ( n ) − ⌈ f ( n ) ⌉ ) || < ǫ } ⊃ n n ∈ Z : || f ( n ) || < ǫ | c | o , it implies that − c ( f − ⌈ f ⌉ ) ∈ GP ′ . By Proposition 4.11 again we conclude that c ⌈ f ⌉ ∈ GP ′ . Hence f ∈ GP ′ . Thus GP ⊆ GP ′ and we are done for the case d = 1by Proposition 4.11 (3).Assume that we have proved F GP d − ⊂ F SGP d − d ≥
2, i.e. the assumption (4.2)holds. We define GP d ( k ) with k = 0 , , , . . . . First f ∈ GP d (0) if and only ifthere is no ⌈ ⌉ in f , i.e. f is the usual polynomial of degree ≤ d . Inductively if GP d (0) , . . . , GP d ( k ) have been defined then f ∈ GP k +1 if and only if f ∈ GP d \ ( S kj =0 GP d ( j )) and there are k + 1 ⌈ ⌉ in f . It is clear that GP d = ∪ ∞ k =0 GP d ( k ). Wenow show GP d ( k ) ⊆ GP ′ d by induction on k . . Huang, S. Shao and X.D. Ye 19 Let f be a usual polynomial of degree ≤ d . Then f ( n ) = a n d + f ( n ) ≃ d a n d with f ∈ GP d − . By Lemma 4.12, f ∈ GP ′ d since a n d ∈ SGP d ⊂ GP ′ d . This shows GP d (0) ⊂ GP ′ d . Now assume that for some k ∈ Z + we have proved(4.3) k [ i =0 GP d ( i ) ⊆ GP ′ d . Let f ∈ GP d ( k + 1). We are going to show that f ∈ GP ′ d . If f = f + f with f , f ∈ S ki =0 GP d ( i ), then by the assumption (4.3) and Proposition 4.11 (2) weconclude that f ∈ GP ′ d . The remaining case is that f can be expressed as the sumof a function in GP d (0) and a function g ∈ GP d ( k + 1) having the form of(1) g = c ⌈ f ⌉ . . . ⌈ f l ⌉ with c = 0, l ≥ g = g ( n ) ⌈ g ( n ) ⌉ . . . ⌈ g l ( n ) ⌉ for any n ∈ Z with g ( n ) ∈ SGP r and r < d .Since GP d (0) ⊂ GP ′ d , f ∈ GP ′ d if and only if g ∈ GP ′ d by Proposition 4.11. Itremains to show that g ∈ GP ′ d . There are two cases.Case (1): g = c ⌈ f ⌉ . . . ⌈ f l ⌉ with c = 0, l ≥ l = 1, then g = c ⌈ f ⌉ with f ∈ GP d ( k ). By Lemma 4.5 we have c ⌈ f ⌉ = cf − c ( f − ⌈ f ⌉ ). It is clear that cf ∈ GP d ( k ) ⊂ GP ′ d since f ∈ GP d ( k ) ⊂ GP ′ d .For any ǫ > { n ∈ Z : || − c ( f ( n ) − ⌈ f ( n ) ⌉ ) || < ǫ } ⊃ n n ∈ Z : || f ( n ) || < ǫ | c | o , it implies that − c ( f − ⌈ f ⌉ ) ∈ GP ′ d . By Proposition 4.11 again we conclude that g = c ⌈ f ⌉ ∈ GP ′ d .If l ≥
2, using Lemmas 4.3 and 4.7 we get that c ⌈ f ⌉ . . . ⌈ f l ⌉ ≃ d − c ( − l X i ,...,il ∈{ , ∗} ( i ,...,il ) =( ∗ ,..., ∗ ) f i . . . f i l l . Since each term of the right side is in GP d ( k ), g ∈ GP ′ d by Lemma 4.12, the assump-tion (4.3) and Proposition 4.11 (2).Case (2): g = g ( n ) ⌈ g ( n ) ⌉ . . . ⌈ g l ( n ) ⌉ for any n ∈ Z with g ∈ SGP r and 1 ≤ r < d .In this case using Lemmas 4.4 and 4.7 we get that g ⌈ g ⌉ . . . ⌈ g l ⌉ ≃ d ( − l X i ,...,il ∈{ , ∗} ( i ,...,il ) =(1 , ∗ ,..., ∗ ) , ( ∗ , ∗ ,..., ∗ ) g i . . . g i l l . Assume i , . . . , i l ∈ { , ∗} with ( i , . . . , i l ) = (1 , ∗ , . . . , ∗ ) , ( ∗ , ∗ , . . . , ∗ ). If there are atleast two 1 appearing in ( i , i , · · · , i l ), then ( − ℓ g i . . . g i ℓ l ∈ S ki =0 GP d ( i ). Hence( − ℓ g i . . . g i ℓ l ∈ GP ′ d by the assumption (4.3). The remaining situation is that i = ∗ and there is exactone 1 appearing in ( i , . . . , i l ). In this case, ( − ℓ g i . . . g i ℓ l ∈ GP d ( k + 1) is the finitesum of the forms a n t ⌈ h ( n ) ⌉ . . . ⌈ h l ′ ( n ) ⌉ with t ≥ h ( n ) = g ( n ); or theforms c ⌈ h l ⌉ . . . ⌈ h l ⌉ or terms in GP ′ d . sets If the term has the form a n t ⌈ h ( n ) ⌉ . . . ⌈ h l ′ ( n ) ⌉ with t ≥ h ( n ) = g ( n ),we let g (1)1 ( n ) = a n t ⌈ h ( n ) ⌉ = a n t ⌈ g ( n ) ⌉ ∈ SGP r . It is clear d ≥ r > r . If r = d , then a n t ⌈ h ( n ) ⌉ . . . ⌈ h l ′ ( n ) ⌉ = g ( n ) ∈ GP ′ d since SGP d ⊂ GP ′ d . If r < d ,then we write a n t ⌈ h ( n ) ⌉ . . . ⌈ h l ′ ( n ) ⌉ = g (1)1 ( n ) ⌈ g (1)2 ( n ) ⌉ . . . ⌈ g (1) l ( n ) ⌉ . By using Case (1) we conclude that g ≃ d finite sum of the forms g (1)1 ( n ) ⌈ g (1)2 ( n ) ⌉ . . . ⌈ g (1) l ( n ) ⌉ and terms in GP ′ d .Repeating the above process finitely many time (at most d -times) we get that g ≃ d finite sum of terms in GP ′ d . Thus g ∈ GP ′ d by Lemma 4.12 and Proposition4.11 (2). The proof is now finished. (cid:3) Proof of Theorem B(1)
In this section, we will prove Theorem B(1), i.e. we will show that if A ∈ F d, then there are k ∈ N , P i ∈ GP d (1 ≤ i ≤ k ) and ǫ i > A ⊃ k \ i =1 { n ∈ Z : P i ( n ) (mod Z ) ∈ ( − ǫ i , ǫ i ) } . We remark that by Section 3.2, it is sufficient to consider the case when the group G is a connected, simply-connected d -step nilpotent Lie group.5.1. Notations.
Let X = G/ Γ with G a connected, simply-connected d -step nilpo-tent Lie group, Γ a uniform subgroup. Let T : X −→ X be the nilrotation inducedby a ∈ G .We assume g is the Lie algebra of G over R , and exp : g −→ G is the exponentialmap. Consider g = g (1) ) g (2) ) . . . ) g ( d ) ) g ( d +1) = { } . We note that [ g ( i ) , g ( j ) ] ⊂ g ( i + j ) . There is a Mal’cev basis X = { X , . . . , X m } for g with(1) For each j = 0 , . . . , m − η j := Span( X j +1 , . . . , X m ) is a Liealgebra ideal in g , and hence H j := exp η j is a normal Lie subgroup of G .(2) For every 0 < i < d we have G i = H l i − +1 .(3) Each g ∈ G can be written uniquely as exp( t X )exp( t X ) . . . exp( t m X m ),for t i ∈ R .(4) Γ consists precisely of those elements which, when written in the above form,have all t i ∈ Z ,where G = G , G i +1 = [ G i , G ] with G d +1 = { e } . Thus, there are 0 = l < l < . . . Define o (0) = 0 and o ( i ) = j if l j − + 1 ≤ i ≤ l j , ≤ j ≤ d − . Huang, S. Shao and X.D. Ye 21 Some lemmas. We need several lemmas. Note that ifexp( t X ) . . . exp( t m X m ) = exp( u X + . . . + u m X m )it is known that [10, 26] each t i is a polynomial of u , . . . , u m and each u i is apolynomial of t , . . . , t m . For our purpose we need to know the precise degree of thepolynomials. Lemma 5.2. Let { X , . . . , X m } be a Mal’cev bases for G/ Γ . Assume thatexp ( t X ) . . . exp ( t m X m ) = exp ( u X + . . . + u m X m ) . Then we have (1) u i = t i for ≤ i ≤ l and if l j − + 1 ≤ i ≤ l j , ≤ j ≤ d then u i = t i + X k o (1)+ ... + kmo ( m ) ≤ o ( i ) ,k ≤ m − , ≤ k ,...,km ≤ m c k ,...,k m ,i t k . . . t k m m , where k is the number of ′ s appearing in { k , . . . , k m } . (2) t i = u i for ≤ i ≤ l and if l j − + 1 ≤ i ≤ l j , ≤ j ≤ d then t i = u i + X k o (1)+ ... + kmo ( m ) ≤ o ( i ) ,k ≤ m − , ≤ k ,...,km ≤ m d k ,...,k m ,i u k . . . u k m m , where k is the number of ′ s appearing in { k , . . . , k m } .Proof. (1). It is easy to see that if m = 1 then d = 1 and (1) holds. So we mayassume that m ≥ 2. For s ∈ { , , . . . , m } { ,...,m } , let { i < . . . < i n } be the collectionof p ′ s with s ( p ) = 0. Let X s = [ X s ( i ) , [ X s ( i ) , . . . , [ X s ( i n − ) , X s ( i n ) ]]] . For each 0 ≤ p ≤ m let k p ( s ) be the number of p ′ s appearing in s (as usual, thecardinality of the empty set is defined as 0). Using the CBH formula m − g ( d +1) = { } it is easy to see that ( t X ) ∗ . . . ∗ ( t m X m ) is the sumof P mi =1 t i X i and the terms constant × t q . . . t q n [ X q , [ X q , . . . , [ X q n − , X q n ]]] , m ≥ n ≥ , i.e. exp( t X ) . . . exp( t m X m ) can be written asexp( m X j =1 t j X j + X s ∈{ , ,...,m }{ ,...,m } k s ) ≤ m − c ′ s t k ( s )1 . . . t k m ( s ) m X s ) . Note that X s ⊂ g ( P mj =1 k j ( s ) o ( j )) . Let X s = P mj =1 c ′ s,j X j . Thus, c ′ s, , . . . , c ′ s,i = 0 if P mj =1 k j ( s ) o ( j ) > o ( i ). Thus, u i = t i for 1 ≤ i ≤ l and if l j − +1 ≤ i ≤ l j , ≤ j ≤ d then the coefficient of X i is u i = t i + X k o (1)+ ... + kmo ( m ) ≤ o ( i ) ,k ≤ m − , ≤ k ,...,km ≤ m c k ,...,k m ,i t k . . . t k m m . Note that when k o (1) + . . . + k m o ( m ) ≤ o ( i ) and k ≤ m − 2, we have that k i = k i +1 = . . . = k m = 0 and some other restrictions. For example, when l + 1 ≤ i ≤ l , sets t k . . . t k m m = t i t i with 1 ≤ i , i ≤ l ; and when l + 1 ≤ i ≤ l , t k . . . t k m m = t i t i t i with 1 ≤ i , i , i ≤ l or t i t i with 1 ≤ i ≤ l and l + 1 ≤ i ≤ l .(2) It is easy to see that t i = u i for 1 ≤ i ≤ l . If d = 1 (2) holds, and thus weassume that d ≥ 2. We show (2) by induction. We assume that(5.1) t p = u p + X k ′ o (1)+ ... + k ′ mo ( m ) ≤ o ( p ) ,k ′ ≤ m − , ≤ k ′ ,...,k ′ m ≤ m d k ′ ,...,k ′ m ,p u k ′ . . . u k ′ m m , for all p with l + 1 ≤ p ≤ i. Since u i +1 = t i +1 + X k o (1)+ ... + kmo ( m ) ≤ o ( i +1) ,k ≤ m − , ≤ k ,...,km ≤ m c k ,...,k m ,i +1 t k . . . t k m m , we have that t i +1 = u i +1 − X k o (1)+ ... + kmo ( m ) ≤ o ( i +1) ,k ≤ m − , ≤ k ,...,km ≤ m c k ,...,k m ,i +1 t k . . . t k m m . Since o ( i + 1) ≤ o ( i ) + 1 and k ≤ m − k o (1) + . . . + k m o ( m ) ≤ o ( i + 1) then k p o ( p ) ≤ o ( i ) for each 1 ≤ p ≤ m , which implies that k i +1 , . . . , k m = 0.By the induction each t p (1 ≤ p ≤ i ) is a polynomial of u , . . . , u m of degree at most P mj =1 k ′ j ≤ o ( p )(see Equation (5.1)) thus X k o (1)+ ... + kmo ( m ) ≤ o ( i +1) ,k ≤ m − , ≤ k ,...,km ≤ m c k ,...,k m ,i +1 t k . . . t k m m is a polynomial of u , . . . , u m of degree at most P mp =1 k p ≤ o ( i + 1) . Rearranging thecoefficients we get (2). Note that k ≤ m − (cid:3) Lemma 5.3. Assume that x = exp ( x X + · · · + x m X m ) and y = exp ( y X ) . . . exp ( y m X m ) . Then xy − = exp ( l X i =1 ( x i − y i ) X i + m X i = l +1 (( x i − y i ) + P i, ( { y p } ) + P i, ( { x p } , { y p } )) X i ) , where P i, ( { y p } ) , P i, ( { x p } , { y p } ) are polynomials of degree at most o ( i ) . Proof. By Lemma 5.2 we have xy − = exp( X )exp( Y )where X = P mi =1 x i X i and Y = − m X i =1 y i X i − m X i = l +1 ( X k ′ o (1)+ ... + k ′ mo ( m ) ≤ o ( i ) ,k ′ ≤ m − , ≤ k ′ ,...,k ′ m ≤ m c k ′ ,...,k ′ m ,i y k ′ . . . y k ′ m m ) X i . . Huang, S. Shao and X.D. Ye 23 Using the CBH formula we get that xy − = exp( X ∗ Y ) = exp( X + Y + 12 [ X, Y ] + 112 [ X, [ X, Y ]] + · · · )= exp( m X i =1 ( x i − y i ) X i + m X i = l +1 ( P i, ( { y p } ) + P i, ( { x p } , { y p } )) X i )= exp( l X i =1 ( x i − y i ) X i + m X i = l +1 (( x i − y i ) + P i, ( { y p } ) + P i, ( { x p } , { y p } )) X i )where P i, ( { y p } ) = − X k ′ o (1)+ ... + k ′ mo ( m ) ≤ o ( i ) ,k ′ ≤ m − , ≤ k ′ ,...,k ′ m ≤ m c k ′ ,...,k ′ m ,i y k ′ . . . y k ′ m m , and P i, ( { x p } , { y p } ) = X P mj =1( kj + k ′ j ) o ( j ) ≤ o ( i ) , ≤ k ,...,km,k ′ ,...,k ′ m ≤ mk ≤ m − ,k ′ ≤ m − e k ′ ,...,k ′ m k ,...,k m x k . . . x k m m y k ′ . . . y k ′ m m . Note that the reason P i, has the above form follows from the fact that [ g ( i ) , g ( j ) ] ⊂ g ( i + j ) , g ( d +1) = { } and a discussion similar to the one used in Lemma 5.2. (cid:3) Proof of Theorem B(1). Let X = G/ Γ with G a connected, simply-connected d -step nilpotent Lie group, Γ a uniform subgroup. Let T : X −→ X be the nilro-tation induced by a ∈ G . Assume that A ⊃ N ( x Γ , U ) with x ∈ X , x Γ ∈ U and U ⊂ G/ Γ open. By Proposition 3.4 we may assume that U is an open neighborhoodof e Γ in G/ Γ, where e is the unit element of G , i.e. A ⊃ N ( e Γ , U ).Assume that a = exp( a X + · · · + a m X m ), where a , · · · a m ∈ R . Then a n = exp( na X + . . . + na m X m )for any n ∈ Z . For h = exp( h X ) . . . exp( h m X m ), where h , · · · , h m ∈ R , write a n h − = exp( p X + . . . + p m X m ) = exp( w X ) . . . exp( w m X m ) . Then by Lemma 5.3 we have p i = na i − h i for 1 ≤ i ≤ l and if l j − + 1 ≤ i ≤ l j , ≤ j ≤ d then(5.2) p i = na i − h i + P i, ( { h p } ) + X P mp =1( kp + k ′ p ) o ( p ) ≤ o ( i ) , ≤ k ,...,km,k ′ ,...,k ′ m ≤ mk ≤ m − ,k ′ ≤ m − e k p ,k ′ p n k + ... + k m h k ′ . . . h k ′ m m , where P i, ( { h p } ) = − P k ′ o (1)+ ... + k ′ mo ( m ) ≤ o ( i ) ,k ′ ≤ m − , ≤ k ′ ,...,k ′ m ≤ m c k ′ ,...,k ′ m ,i h k ′ . . . h k ′ m m and e k p ,k ′ p = e k ′ ,...,k ′ m k ,...,k m a k . . . a k m m . sets Changing the exponential coordinates to Mal’sev coordinates (Lemma 5.2), weget that w i = na i − h i for 1 ≤ i ≤ l and if l j − + 1 ≤ i ≤ l j , ≤ j ≤ d then w i = p i + X k o (1)+ ... + kmo ( m ) ≤ o ( i ) ,k ≤ m − , ≤ k ,...,km ≤ m d k ,...,k m ,i p k . . . p k m m , in this case using (5.2) it is not hard to see that w i is the sum of − h i and Q i = Q i ( n, h , . . . , h i − ) such that Q i is the sum of terms c ( k, k , . . . , k i − ) n k h k . . . h k i − i − with k + k o (1) + . . . + k i − o ( i − ≤ o ( i ) (see the argument of Lemma 5.2(2)). Notethat if k = 0 then k ≤ m − 2, and if k = . . . = k m = 0 then k ≥ n ∈ Z , let h i ( n ) = ⌈ na i ⌉ if 1 ≤ i ≤ l , and let h i ( n ) = ⌈ Q i ( n, h ( n ) , . . . , h i − ( n )) ⌉ if l j − + 1 ≤ i ≤ l j , ≤ j ≤ d . Again a similar argument as in the proof of Lemma5.2(2) shows that h i ( n ) is well defined and is a generalized polynomial of degree ofmost o ( i ) ≤ d . For example, if l + 1 ≤ i ≤ l then p i = na i − h i + X ≤ i
Proof of Theorem B(2) In this section, we aim to prove Theorem B(2), i.e. F d, ⊃ F GP d . To do this first wemake some preparations, then derive some results under the inductive assumption,and finally give the proof. Note that in the construction the nilpotent matrix Liegroup is used.More precisely, to show F d, ⊃ F GP d we need only to prove F d, ⊃ F SGP d byTheorem B. To do this, for a given F ∈ F SGP d we need to find a d -step nilsystem( X, T ), x ∈ X and a neighborhood U of x such that F ⊃ N ( x , U ). In theprocess doing this, we find that it is convenient to consider a finite sum of speciallygeneralized polynomials P ( n ; α , . . . , α r ) (defined in (6.4)) instead of considering asingle specially generalized polynomial. We can prove that F d, ⊃ F GP d if and only if { n ∈ Z : || P ( n ; α , · · · , α d ) || < ǫ } ∈ F d, for any α , · · · , α d ∈ R and ǫ > X, T ) as the closure of the orbit of Γ in G d / Γ (the nilrotation isinduced by a matrix A ∈ G d ), and consider the most right-corner entry z d ( n ) in A n B n with B n ∈ Γ. We finish the proof by showing that P ( n ; α , · · · , α d ) ≃ d z d ( n )and { n ∈ Z : || z d ( n ) || < ǫ } ∈ F d, for any ǫ > sets Some preparations. For a matrix A in G d we now give a precise formula of A n . Lemma 6.1. Let x = ( x ki ) ≤ k ≤ d, ≤ i ≤ d − k +1 ∈ R d ( d +1) / . For n ∈ N , assume that x ( n ) = ( x ki ( n )) ≤ k ≤ d, ≤ i ≤ d − k +1 ∈ R d ( d +1) / satisfies M ( x ( n )) = M ( x ) n , then (6.1) x ki ( n ) = (cid:0) n (cid:1) P ( x ; i, k ) + (cid:0) n (cid:1) P ( x ; i, k ) + · · · + (cid:0) nk (cid:1) P k ( x ; i, k ) for ≤ k ≤ d and ≤ i ≤ d − k + 1 , where (cid:0) nk (cid:1) = n ( n − ··· ( n − k +1) k ! for n, k ∈ N and P ℓ ( x ; i, k ) = X ( s ,s , ··· ,sℓ ) ∈{ , , ··· ,k } ℓs s ··· + sℓ = k x s i x s i + s x s i + s + s · · · x s ℓ i + s + s + ··· + s ℓ − for ≤ k ≤ d , ≤ i ≤ d − k + 1 and ≤ ℓ ≤ k .Proof. Let x i = 1 and x i ( m ) = 1 for 1 ≤ i ≤ d and m ∈ N . By (3.2), it is not hardto see that(6.2) x ki ( m + 1) = k X j =0 x k − ji ( m ) · x ji + k − j for 1 ≤ k ≤ d , 1 ≤ i ≤ d − k + 1 and m ∈ N .Now we do induction for k . When k = 1, x i (1) = x i and x i ( m + 1) = x i ( m ) + x i for m ∈ N by (6.2). Hence x i ( n ) = nx i = (cid:0) n (cid:1) P ( x ; i, ≤ i ≤ d and n ∈ N if k = 1.Assume that 1 ≤ ℓ ≤ d − 1, and (6.1) holds for each 1 ≤ k ≤ ℓ , 1 ≤ i ≤ d − k + 1and n ∈ N . For k = ℓ + 1, we make induction on n . When n = 1 it is clear x ki (1) = x ki = (cid:0) (cid:1) P ( x ; i, k ) + (cid:0) (cid:1) P ( x ; i, k ) + · · · + (cid:0) k (cid:1) P k ( x ; i, k )for 1 ≤ i ≤ d − k + 1. That is, (6.1) holds for k = ℓ + 1, 1 ≤ i ≤ d − k + 1 and n = 1.Assume for n = m ≥ 1, (6.1) holds for k = ℓ + 1, 1 ≤ i ≤ d − k + 1 and n = m . For n = m + 1, by (6.2) x ki ( n ) = x ki ( m ) + (cid:16) k − X j =1 x k − ji ( m ) · x ji + k − j (cid:17) + x ki = x ki ( m ) + (cid:16) k − X j =1 ( k − j X r =1 (cid:0) mr (cid:1) P r ( x ; i, k − j )) · x ji + k − j ) (cid:17) + x ki = x ki ( m ) + (cid:16) k − X r =1 ( k − r X j =1 P r ( x ; i, k − j ) x ji + k − j ) (cid:0) mr (cid:1)(cid:17) + x ki = x ki ( m ) + (cid:16) k − X r =1 ( k − X j = r P r ( x ; i, j ) x k − ji + j ) (cid:0) mr (cid:1)(cid:17) + x ki for 1 ≤ i ≤ d − k + 1. Note that . Huang, S. Shao and X.D. Ye 27 k − X j = r P r ( x ; i, j ) x k − ji + j = k − X j = r X ( s , ··· ,sr ) ∈{ , , ··· ,k − } rs ··· + sr = j x s i x s i + s · · · x s r i + s + ··· + s r − x k − ji + j which is equal to X ( s , ··· ,sr,sr +1) ∈{ , , ··· ,k − } r +1 s s ··· + sr + sr +1= k x s i x s i + s · · · x s r i + s + ··· + s r − x s r +1 i + s + ··· + s r − + s r = P r +1 ( x ; i, k )for 1 ≤ r ≤ k − ≤ i ≤ d − k + 1. Collecting terms we have x ki ( n ) = x ki ( m ) + (cid:16) k − X r =1 P r +1 ( x ; i, k ) (cid:0) mr (cid:1)(cid:17) + x ki = x ki ( m ) + (cid:16) k X r =2 P r ( x ; i, k ) (cid:0) mr − (cid:1)(cid:17) + P ( x ; i, k )= (cid:16) m X r =1 P r ( x ; i, k ) (cid:0) mr (cid:1)(cid:17) + (cid:16) k X r =2 P r ( x ; i, k ) (cid:0) mr − (cid:1)(cid:17) + P ( x ; i, k ) . Rearranging the order we get x ki ( n ) = ( m + 1) P ( x ; i, k ) + k X r =2 (cid:16)(cid:0) mr (cid:1) + (cid:0) mr − (cid:1)(cid:17) P r ( x ; i, k )= k X r =1 (cid:0) m +1 r (cid:1) P r ( x ; i, k ) = k X r =1 (cid:0) nr (cid:1) P r ( x ; i, k )for 1 ≤ i ≤ d − k + 1. This ends the proof of the lemma. (cid:3) Remark . By the above lemma, we have P ( x ; i, k ) = x ki and P k ( x ; i, k ) = x i x i +1 · · · x i + k − for 1 ≤ k ≤ d and 1 ≤ i ≤ d − k + 1.6.2. Consequences under the inductive assumption. We will use induction toshow Theorem B(2). To make the proof clearer, we derive some results under thefollowing inductive assumption.(6.3) F d − , ⊃ F GP d − , where d ∈ N with d ≥ 2. For that purpose, we need more notions and lemmas.The proof of Lemma 6.3 is similar to the one of Lemma 4.7, where W d is defined inDefinition 4.6. Lemma 6.3. Under the assumption (6.3) , one has for any p ( n ) ∈ W d and ǫ > , { n ∈ Z : p ( n ) ( mod Z ) ∈ ( − ǫ, ǫ ) } ∈ F d − , . sets Definition 6.4. For r ∈ N , we define g GP r = { p ( n ) ∈ GP r : { n ∈ Z : p ( n ) (mod Z ) ∈ ( − ǫ, ǫ ) } ∈ F r, for any ǫ > } . Remark . It is clear that for p ( n ) ∈ GP r , p ( n ) ∈ g GP r if and only if − p ( n ) ∈ g GP r .Since F r, is a filter, if p ( n ) , p ( n ) , · · · , p k ( n ) ∈ g GP r then p ( n ) + p ( n ) + · · · + p k ( n ) ∈ g GP r . Moreover by the definition of g GP d , we know that F d, ⊃ F GP d if and only if g GP d = GP d . Lemma 6.6. Let p ( n ) , q ( n ) ∈ GP d with p ( n ) ≃ d q ( n ) . Under the assumption (6.3) , p ( n ) ∈ g GP d if and only if q ( n ) ∈ g GP d .Proof. This follows from Lemma 6.3 and the fact that F d, is a filter. (cid:3) For α , α , . . . , α r ∈ R , r ∈ N , we define P ( n ; α , α , · · · , α r )= r X ℓ =1 X j , ··· ,jℓ ∈ N j ··· + jℓ = r ( − ℓ − L (cid:16) n j j ! j Y r =1 α r , n j j ! j Y r =1 α j + r , · · · , n j ℓ j ℓ ! j ℓ Y r ℓ =1 α ℓ − P t =1 j t + r ℓ (cid:17) (6.4)where the definition of L is given in (4.1). Theorem 6.7. Under the assumption (6.3) , the following properties are equivalent: (1) F d, ⊃ F GP d . (2) P ( n ; α , α , · · · , α d ) ∈ g GP d for any α , α , · · · , α d ∈ R , that is { n ∈ Z : P ( n ; α , α , · · · , α d ) ( mod Z ) ∈ ( − ǫ, ǫ ) } ∈ F d, for any α , α , · · · , α d ∈ R and ǫ > . (3) SGP d ⊂ g GP d .Proof. (1) ⇒ (2). Assume F d, ⊃ F GP d . By the definition of g GP d , we know that F d, ⊃ F GP d if and only if g GP d = GP d . Particularly P ( n ; α , α , · · · , α d ) ∈ g GP d forany α , α , · · · , α d ∈ R .(3) ⇒ (1). Assume that SGP d ⊂ g GP d . Then F d, ⊇ F SGP d . Moveover F d, ⊃F SGP d = F GP d by Theorem 4.13.(2) ⇒ (3). Assume that P ( n ; α , α , · · · , α d ) ∈ g GP d for any α , α , · · · , α d ∈ R .We defineΣ d = { ( j , j , · · · , j ℓ ) : ℓ ∈ { , , · · · , d } , j , j , · · · , j ℓ ∈ N and ℓ X t =1 j t = d } . For ( j , j , · · · , j ℓ ) , ( r , r , · · · , r s ) ∈ Σ d , we say ( j , j , · · · , j ℓ ) > ( r , r , · · · , r s ) ifthere exists 1 ≤ t ≤ ℓ such that j t > r s and j i = r i for i < t . Clearly (Σ d , > )is a totally ordered set with the maximal element ( d ) and the minimal element(1 , , · · · , . Huang, S. Shao and X.D. Ye 29 For j = ( j , j , · · · , j ℓ ) ∈ Σ d , put L ( j ) = { L ( n j a , · · · , n j ℓ a ℓ ) : a , · · · , a ℓ ∈ R } . Now, we have Claim: L ( s ) ⊆ g GP d for each s ∈ Σ d . Proof. We do induction for s under the order > . First, consider the case when s = ( d ). Given a ∈ R , we take α = 1 , α = 2 , · · · α d − = d − α d = da . Thenfor any 1 ≤ j ≤ d − n j j ! j Q t =1 α t ∈ Z for n ∈ Z . Thus P ( n ; α , α , · · · , α d ) = L ( n d d ! d Y t =1 α t ) = L ( n d a ) (mod Z )for any n ∈ Z . Hence L ( n d a ) ∈ g GP d since P ( n ; α , α , · · · , α d ) ∈ g GP d . Since a isarbitrary, we conclude that L (( d )) ⊂ g GP d .Assume that for any s > i = ( i , · · · , i k ) ∈ Σ d , we have L ( s ) ⊂ g GP d . Nowconsider the case when s = i = ( i , · · · , i k ). There are two cases.The first case is k = d , i = i = · · · = i d = 1. Given a , a , · · · , a d ∈ R , bythe assumption we have that for any ( j , j , · · · , j ℓ ) > i , L (( j , j , · · · , j ℓ )) ⊂ g GP d .Thus d − X ℓ =1 X j , ··· ,jℓ ∈ N j ··· + jℓ = r ( − ℓ − L (cid:16) n j j ! j Y r =1 a r , n j j ! j Y r =1 a j + r , · · · , n j ℓ j ℓ ! j ℓ Y r ℓ =1 a P ℓ − t =1 j t + r ℓ (cid:17) belongs to g GP d by the Remark 6.5. This implies P ( n ; a , a , · · · , a d ) − L ( na , na , · · · , na d ) ∈ g GP d by (6.4). Combining this with P ( n ; a , a , · · · , a d ) ∈ g GP d , we have L ( na , na , · · · , na d ) ∈ g GP d by Remark 6.5. Since a , a , · · · , a d ∈ R are arbitrary, we get L ( i ) ⊂ g GP d .The second case is i > (1 , , · · · , a , a , · · · , a k ∈ R , for r = 1 , , · · · , k ,we put α P r − t =1 i t + h = h for 1 ≤ h ≤ i r − α P r − t =1 i t + i r = i r a r .By the assumption, for ( j , j , · · · , j ℓ ) > i , L (cid:16) n j j ! j Y r =1 α r , n j j ! j Y r =1 α j + r , · · · , n j ℓ j ℓ ! j ℓ Y r ℓ =1 α P ℓ − t =1 j t + r ℓ (cid:17) ∈ g GP d . For ( j , j , · · · , j ℓ ) < i , there exists 1 ≤ u ≤ k such that j t = i t for 1 ≤ t ≤ u − i u > j u . Then(6.5) n j u j u ! j u Y r u =1 α P u − t =1 j t + r u = n j u . sets When u = 1, by (6.5), L (cid:16) n j j ! j Y r =1 α r , · · · , · · · , n j ℓ j ℓ ! j ℓ Y r ℓ =1 α P ℓ − t =1 j t + r ℓ (cid:17) ∈ Z for any n ∈ Z . Hence L (cid:16) n j j ! j Y r =1 α r , · · · , · · · , n j ℓ j ℓ ! j ℓ Y r ℓ =1 α P ℓ − t =1 j t + r ℓ (cid:17) ∈ g GP d . When u > 1, write β v = Q j v r v =1 α P v − t =1 j t + r v for v = 1 , , · · · , ℓ . Then β u = 1 and ⌈ L ( n j u β u , n j u +1 β u +1 · · · , n j ℓ β ℓ ) ⌉ = L ( n j u β u , n j u +1 β u +1 · · · , n j ℓ β ℓ ) . Moreover, L (cid:16) n j j ! j Y r =1 α r , · · · , n j u j u ! j u Y r u =1 α P u − t =1 j t + r ℓ , · · · , n j ℓ j ℓ ! j ℓ Y r ℓ =1 α P ℓ − t =1 j t + r ℓ (cid:17) is equal to L (cid:0) n j β , · · · , n j u β u , · · · , n j ℓ β ℓ (cid:1) = L (cid:0) n j β , · · · , n j u − β u − ⌈ L ( n j u β u , · · · , n j ℓ β ℓ ) ⌉ (cid:1) which is equal to L (cid:0) n j β , · · · , n j u − β u − L ( n j u β u , n j u +1 β u +1 · · · , n j ℓ β ℓ ) (cid:1) = L (cid:0) n j β , · · · , n j u − + j u β u − β u ⌈ L ( n j u +1 β u +1 · · · , n j ℓ β ℓ ) ⌉ (cid:1) = L (cid:0) n j β , · · · , n j u − + j u β u − β u , n j u +1 β u +1 · · · , n j ℓ β ℓ (cid:1) ∈ g GP d since ( j , · · · , j u − , j u − + j u , j u +1 , · · · , j ℓ ) > i .Summing up for any j = ( j , · · · , j ℓ ) ∈ Σ d with j = i , we have L (cid:16) n j j ! j Y r =1 α r , · · · , n j u j u ! j u Y r u =1 α P u − t =1 j t + r ℓ , · · · , n j ℓ j ℓ ! j ℓ Y r ℓ =1 α P ℓ − t =1 j t + r ℓ (cid:17) ∈ g GP d . Combining this with P ( n ; α , · · · , α d ) ∈ g GP d , we have L (cid:16) n i a , n i a , · · · , n i k a k (cid:17) = L (cid:16) n i i ! i Y r =1 α r , n i i ! i Y r =1 α i + r , · · · , n i k i k ! i k Y r k =1 α P k − t =1 i t + r k (cid:17) ∈ g GP d by (6.4) and Remark (6.5). Since a , · · · , a k ∈ R are arbitrary, L ( i ) ⊂ g GP d . (cid:3) Finally, since SGP d = S j ∈ Σ d L ( j ), we have SGP d ⊂ g GP d by the above Claim. (cid:3) . Huang, S. Shao and X.D. Ye 31 Proof of Theorem B(2). We are now ready to give the proof of the TheoremB(2). As we said before, we will use induction to show Theorem B(2). Firstly, for d = 1, since F GP = F SGP and F , is a filter, it is sufficient to show for any a ∈ R and ǫ > { n ∈ Z : an (mod Z ) ∈ ( − ǫ, ǫ ) } ∈ F , . This is obvious since the rotation on the unit circle is a 1-step nilsystem.Now we assume that F d − , ⊃ F GP d − , i.e. the the assumption (6.3) holds. ByTheorem 6.7, to show F d, ⊃ F GP d , it remains to prove that P ( n ; α , α , · · · , α d ) ∈ g GP d for any α , α , . . . , α d ∈ R , that is { n ∈ Z : P ( n ; α , α , · · · , α d ) (mod Z ) ∈ ( − ǫ, ǫ ) } ∈ F d, for any α , α , · · · , α d ∈ R and ǫ > α , α , · · · , α d ∈ R and choose x = ( x ki ) ≤ k ≤ d, ≤ i ≤ d − k +1 ∈ R d ( d +1) / with x i = α i for i = 1 , , · · · , d and x ki = 0 for 2 ≤ k ≤ d and 1 ≤ i ≤ d − k + 1. Then A = M ( x ) = α . . . α . . . . . . α d − 00 0 0 . . . α d . . . For n ∈ N , if x ( n ) = ( x ki ( n )) ≤ k ≤ d, ≤ i ≤ d − k +1 ∈ R d ( d +1) / satisfies M ( x ( n )) = A n ,then by Lemma 6.1 and Remark 6.2,(6.6) x ki ( n ) = (cid:0) nk (cid:1) P k ( x ; i, k ) = (cid:0) nk (cid:1) x i x i +1 · · · x i + k − = (cid:0) nk (cid:1) α i α i +1 · · · α i + k − for 1 ≤ k ≤ d and 1 ≤ i ≤ d − k + 1.Now we define f i ( n ) = ⌈ x i ( n ) ⌉ = ⌈ nα i ⌉ for 1 ≤ i ≤ d and inductively for k = 2 , , · · · , d define(6.7) f ki ( n ) = (cid:24) x ki ( n ) − k − X j =1 x k − ji ( n ) f ji + k − j ( n ) (cid:25) for 1 ≤ i ≤ d − k + 1. Then we define z i ( n ) = x i ( n ) − f i ( n )for 1 ≤ i ≤ d and inductively for k = 2 , , · · · , d define(6.8) z ki ( n ) = x ki ( n ) − (cid:16) k − X j =1 x k − ji ( n ) f ji + k − j ( n ) (cid:17) − f ki ( n )for 1 ≤ i ≤ d − k + 1.It is clear that z ki ( n ) ∈ GP k for 1 ≤ k ≤ d and 1 ≤ i ≤ d − k + 1. First, we have Claim: P ( n ; α , α , · · · , α d ) ≃ d z d ( n ).Since the proof of the Claim is long, the readers find the proof in the followingsubsection. Now we are going to show z d ( n ) ∈ g GP d . sets Let X = G d / Γ be endowed with the metric ρ in Lemma 3.7 and T be the nil-rotation induced by A ∈ G d , i.e. B Γ AB Γ for B ∈ G d . Since G d is a d -stepnilpotent Lie group and Γ is a uniform subgroup of G d , ( X, T ) is a d -step nilsystem.Let x = Γ ∈ X and Z be the closure of the orbit orb( x , T ) of e in X . Then ( Z, T )is a minimal d -step nilsystem. We consider ρ as a metric on Z .For a given η > δ > e δ + δ + ··· + δ d − < min { , η } . Put U = { z ∈ Z : ρ ( z, x ) < δ } and S = { n ∈ N : ρ ( A n Γ , Γ) < δ } = { n ∈ Z : T n x ∈ U } . Then S ∈ F d, since ( Z, T ) is a minimal d -step nilsystem. In the following we aregoing to show that { m ∈ Z : z d ( m ) (mod Z ) ∈ ( − η, η ) } ⊃ S. This clearly implies that { m ∈ Z : z d ( m ) (mod Z ) ∈ ( − η, η ) } ∈ F d, since S ∈ F d, .As η > z d ( n ) ∈ g GP d .Given n ∈ S , one has ρ ( A n Γ , Γ) < δ . Since ρ is right-invariant and Γ is a group,there exists B − n ∈ Γ such that ρ ( A n , B − n ) < δ . Take h ( n ) = ( − h ki ( n )) ≤ k ≤ d, ≤ i ≤ d − k +1 ∈ Z d ( d +1) / with M ( h ( n )) = B n . By (3.3),(6.9) k A n B n − I k ≤ e δ + δ + ··· + δ d − < min n , η o . Let y ( n ) = ( y ki ( n )) ≤ k ≤ d, ≤ i ≤ d − k +1 ∈ R d ( d +1) / such that M ( y ( n )) = A n B n = M ( x ( n )) M ( h ( n )) . By (3.2)(6.10) y ki ( n ) = x ki ( n ) − (cid:16) k − X j =1 x k − ji ( n ) h ji + k − j ( n ) (cid:17) − h ki ( n )for 1 ≤ k ≤ d and 1 ≤ i ≤ d − k + 1. Thus(6.11) | y ki ( n ) | < min { , η } for 1 ≤ k ≤ d and 1 ≤ i ≤ d − k + 1 by (6.9). Hence h i ( n ) = ⌈ x i ( n ) ⌉ = ⌈ nα i ⌉ for1 ≤ i ≤ d and(6.12) h ki ( n ) = (cid:24) x ki ( n ) − k − X j =1 x k − ji ( n ) h ji + k − j ( n ) (cid:25) for 2 ≤ k ≤ d and 1 ≤ i ≤ d − k + 1.Since h i ( n ) = ⌈ nα i ⌉ = f i ( n ) for 1 ≤ i ≤ d , one has h ki ( n ) = f ki ( n ) for 2 ≤ k ≤ d and 1 ≤ i ≤ d − k + 1 by (6.7) and (6.12). Moreover by (6.8) and (6.10), we know z ki ( n ) = y ki ( n ) for 2 ≤ k ≤ d and 1 ≤ i ≤ d − k + 1. Combining this with (6.11), | z ki ( n ) | < min { , η } for 1 ≤ k ≤ d and 1 ≤ i ≤ d − k + 1. Particularly, | z d ( n ) | < η .Thus n ∈ { m ∈ Z : z d ( m ) (mod Z ) ∈ ( − η, η ) } , . Huang, S. Shao and X.D. Ye 33 which implies that { m ∈ Z : z d ( m ) (mod Z ) ∈ ( − η, η ) } ⊃ S . That is, z d ( n ) ∈ g GP d .Finally using the Claim and the fact that z d ( n ) ∈ g GP d we have P ( n ; α , α , · · · , α d ) ∈ g GP d by Lemma 6.6. This ends the proof, i.e. we have proved F d, ⊃ F GP d .6.4. Proof of the Claim. Let u ki ( n ) = z ki ( n ) + f ki ( n ) = x ki ( n ) − k − X j =1 x k − ji ( n ) f ji + k − j ( n )for 1 ≤ k ≤ d and 1 ≤ i ≤ d − k + 1. Then f ki ( n ) = ⌈ u ki ( n ) ⌉ for 1 ≤ k ≤ d and 1 ≤ i ≤ d − k + 1.We define U ( n ; j ) = n j j ! Q j r =1 α r for 1 ≤ j ≤ d . Then inductively for ℓ =2 , , · · · , d we define U ( n ; j , j , · · · , j ℓ ) = ( U ( n ; j , · · · , j ℓ − ) − ⌈ U ( n ; j , · · · , j ℓ − ) ⌉ ) n j ℓ j ℓ ! j ℓ Y r =1 α P ℓ − t =1 j t + r = ( U ( n ; j , · · · , j ℓ − ) − ⌈ U ( n ; j , · · · , j ℓ − ) ⌉ ) L ( n j ℓ j ℓ ! j ℓ Y r ℓ =1 α P ℓ − t =1 j t + r ℓ )for j , j , · · · , j ℓ ≥ j + · · · + j ℓ ≤ d (see (4.1) for the definition of L ).Next, U ( n ; d ) = n d d ! d Q r =1 α r = L ( n d d ! d Q r =1 α r ) and for 2 ≤ ℓ ≤ d , j , j , · · · , j ℓ ∈ N with j + j + · · · + j ℓ = d , by Lemma 4.9(1) U ( n ; j , j , · · · , j ℓ ) = ( U ( n ; j , · · · , j ℓ − ) − ⌈ U ( n ; j , · · · , j ℓ − ) ⌉ ) L ( n j ℓ j ℓ ! j ℓ Y r ℓ =1 α P ℓ − t =1 j t + r ℓ ) ≃ d U ( n ; j , · · · , j ℓ − ) ⌈ L ( n j ℓ j ℓ ! j ℓ Y r ℓ =1 α P ℓ − t =1 j t + r ℓ ) ⌉ which is= ( U ( n ; j , · · · , j ℓ − ) − ⌈ U ( n ; j , · · · , j ℓ − ) ⌉ ) × L ( n j ℓ − j ℓ − ! j ℓ − Y r ℓ − =1 α P ℓ − t =1 j t + r ℓ − , n j ℓ j ℓ ! j ℓ Y r ℓ =1 α P ℓ − t =1 j t + r ℓ ) ≃ d U ( n ; j , · · · , j ℓ − ) ⌈ L ( n j ℓ − j ℓ − ! j ℓ − Y r ℓ − =1 α P ℓ − t =1 j t + r ℓ − , n j ℓ j ℓ ! j ℓ Y r ℓ =1 α P ℓ − t =1 j t + r ℓ ) ⌉ . Continuing the above argument we have U ( n ; j , j , · · · , j ℓ ) ≃ d L (cid:16) n j j ! j Y r =1 α r , n j j ! j Y r =1 α j + r , · · · , n j ℓ j ℓ ! j ℓ Y r ℓ =1 α P ℓ − t =1 j t + r ℓ (cid:17) . sets That is, for 1 ≤ ℓ ≤ d , j , j , · · · , j ℓ ∈ N with j + j + · · · + j ℓ = d ,(6.13) U ( n ; j , j , · · · , j ℓ ) ≃ d L (cid:16) n j j ! j Y r =1 α r , n j j ! j Y r =1 α j + r , · · · , n j ℓ j ℓ ! j ℓ Y r ℓ =1 α P ℓ − t =1 j t + r ℓ (cid:17) . Thus using (6.13) we have(6.14) P ( n ; α , α , · · · , α d ) ≃ d d X ℓ =1 X j , ··· jℓ ∈ N j ··· + jℓ = d ( − ℓ − U ( n ; j , j , · · · , j ℓ ) . Next using Lemma 4.9(1), for any j , j , · · · , j ℓ ∈ N with j + j , · · · + j ℓ ≤ d − U ( n ; j , · · · , j ℓ ) f d − P ℓt =1 j t P ℓt =1 j t ( n ) is equal to U ( n ; j , · · · , j ℓ ) ⌈ u d − P ℓt =1 j t P ℓt =1 j t ( n ) ⌉≃ d (cid:16) U ( n ; j , · · · , j ℓ ) − ⌈ U ( n ; j , · · · , j ℓ ) ⌉ (cid:17) u d − P ℓt =1 j t P ℓt =1 j t ( n )= (cid:16) U ( n ; j , · · · , j ℓ ) − ⌈ U ( n ; j , · · · , j ℓ ) ⌉ (cid:17) × (cid:16) x d − P ℓt =1 j t P ℓt =1 j t ( n ) − d − ( P ℓt =1 j t ) − X j ℓ +1 =1 x j ℓ +1 P ℓt =1 j t ( n ) f d − P ℓ +1 t =1 j t P ℓ +1 t =1 j t ( n ) (cid:17) which is equal to (cid:16) U ( n ; j , j , · · · , j ℓ ) − ⌈ U ( n ; j , j , · · · , j ℓ ) ⌉ (cid:17) × (cid:0) nd − ℓ P t =1 j t (cid:1) d − ℓ P t =1 j t Y r ℓ +1 =1 α ℓ P t =1 j t + r ℓ +1 − d − ℓ +1 P t =1 j t − X j ℓ +1 =1 (cid:0) nj ℓ +1 (cid:1) j ℓ +1 Y r ℓ +1 =1 α ℓ P t =1 j t + r ℓ +1 f d − ℓ +1 P t =1 j t ℓ +1 P t =1 j t ( n ) ! which is ≃ d (cid:16) U ( n ; j , j , · · · , j ℓ ) − ⌈ U ( n ; j , j , · · · , j ℓ ) ⌉ (cid:17) × n d − ℓ P t =1 j t ( d − ℓ P t =1 j t )! d − ℓ P t =1 j t Y r ℓ +1 =1 α ℓ P t =1 j t + r ℓ +1 − d − ℓ +1 P t =1 j t − X j ℓ +1 =1 n j ℓ +1 j ℓ +1 ! j ℓ +1 Y r ℓ +1 =1 α ℓ P t =1 j t + r ℓ +1 f d − ℓ +1 P t =1 j t ℓ +1 P t =1 j t ( n ) ! = U ( n ; j , · · · , j ℓ , d − ℓ X t =1 j t ) − d − ℓ +1 P t =1 j t − X j ℓ +1 =1 U ( n ; j , · · · , j ℓ , j ℓ +1 ) f d − ℓ +1 P t =1 j t ℓ +1 P t =1 j t ( n ) . . Huang, S. Shao and X.D. Ye 35 Using the fact and Lemma 4.9(1), we have z d ( n ) ≃ d u d ( n ) = x d ( n ) − d − X j =1 x j ( n ) f d − j j ( n )= (cid:0) nd (cid:1) α α · · · α d − d − X j =1 (cid:0) nj (cid:1) α α · · · α j f d − j j ( n ) ≃ d U ( n ; d ) − d − X j =1 U ( n ; j ) f d − j j ( n ) ≃ d U ( n ; d ) − (cid:16) d − X j =1 ( U ( n ; j , d − j ) − d − j − X j =1 U ( n ; j , j ) f d − ( j + j )1+ j + j ( n )) (cid:17) . Continuing this argument we obtain z d ( n ) ≃ d d X ℓ =1 X j , ··· ,jℓ ∈ N j ··· + jℓ ( − ℓ − U ( n ; j , · · · , j ℓ ) . Combining this with (6.14), we have proved the Claim.7. Proof of Theorem C In this section we will prove Theorem C. That is, we will show that for d ∈ N and F ∈ F GP d , there exist a minimal d -step nilsystem ( X, T ) and a nonempty open set U such that F ⊃ { n ∈ Z : U ∩ T − n U ∩ . . . ∩ T − dn U = ∅} . Let us explain the idea of the proof of Theorem C. Put N d = { B ⊆ Z : thereare a minimal d -step nilsystem ( X, T ) and an open non-empty set U of X with B ⊃ { n ∈ Z : T di =0 T − in U = ∅}} . Similar to the proof of Theorem B(2) we firstshow that F GP d ⊆ N d if and only if { n ∈ Z : || P ( n ; α , · · · , α d ) || < ǫ } ∈ N d for any α , · · · , α d ∈ R and ǫ > 0. We choose ( X, T ) as the closure of the orbit of Γ in G d / Γ(the nilrotation is induced by a matrix A ∈ G d ), define U ⊂ X depending on a given ǫ > 0, put S = { n ∈ Z : T di =0 T − in U = ∅} ; and consider the most right-corner entry z d ( m ) in A nm BC m with B ∈ G d and C m ∈ Γ for a given n ∈ S with 1 ≤ m ≤ d .We finish the proof by showing S ⊂ { n ∈ Z : || P ( n ; α , · · · , α d ) || < ǫ } which impliesthat { n ∈ Z : || P ( n ; α , · · · , α d ) || < ǫ } ∈ N d .7.1. The ordinary polynomial case. To illustrate the idea of the proof of The-orem C, we first consider the situation when the generalized polynomials are theordinary ones. That is, we want to explain if p ( n ) is a polynomial of degree d with p (0) = 0 and ǫ > 0, how we can find a d -step nilsystem ( X, T ), and a nonemptyopen set U ⊂ X such that(7.1) { n ∈ Z : p ( n ) (mod Z ) ∈ ( − ǫ, ǫ ) } ⊃ { n ∈ Z : U ∩ T − n U ∩ . . . ∩ T − dn U = ∅} . sets To do this define T α,d : T d −→ T d by T α,d ( θ , θ , . . . , θ d ) = ( θ + α, θ + θ , θ + θ , . . . , θ d + θ d − ) , where α ∈ R . A simple computation yields that T nα,d ( θ , . . . , θ d ) = ( θ + nα, nθ + θ + 12 n ( n − α, . . . , d X i =0 (cid:0) nd − i (cid:1) θ i ) , (7.2)where θ = α , n ∈ Z and (cid:0) n (cid:1) = 1, (cid:0) ni (cid:1) := Q i − j =0 ( n − j ) i ! for i = 1 , , · · · , d .We now prove (7.1) by induction. The case when d = 1 is easy, and we assumethat for each polynomial of degree ≤ d − p ( n ) = P di =1 α i n i with α i ∈ R . By induction for each 1 ≤ i ≤ d − i -step nilsystem ( X i , T i )and an open non-empty subset U i of X i such that { n ∈ Z : α i n i (mod Z ) ∈ ( − ǫd , ǫd ) } ⊃ { n ∈ Z : U i ∩ T − ni U i ∩ . . . ∩ T − dni U i = ∅} . By the Vandermonde’s formula, we know . . . d . . . d ... ... ... ... ...1 2 d − d − . . . d d − d d . . . d d is a non-singular matrix. Hence there are integers λ , λ , . . . , λ d and λ ∈ N suchthat the following equation holds: . . . d . . . d ... ... ... ... ...1 2 d − d − . . . d d − d d . . . d d λ λ ... λ d − λ d = λ That is, d X m =1 λ m m j = λ + λ j + . . . + λ d d j = 0 , ≤ j ≤ d − d X m =1 λ m m d = λ + λ d + . . . + λ d d d = λ. (7.3)Now let T d = T αdλ ,d and Y d = T d . Let K d = d ! P di =1 | λ i | , ǫ > K d ǫ < ǫ/d and U d = ( − ǫ , ǫ ) d .It is easy to see that if n ∈ { n ∈ Z : U d ∩ T − nd U d ∩ . . . ∩ T − dnd U d = ∅} then we knowthat there is ( θ , . . . , θ d ) ∈ U d such that T ind ( θ , . . . , θ d ) ∈ U d for each 1 ≤ i ≤ d . . Huang, S. Shao and X.D. Ye 37 Thus, by (7.2) considering the last coordinate we ge that (cid:0) nd (cid:1) θ + (cid:0) nd − (cid:1) θ + . . . + (cid:0) n (cid:1) θ d (mod Z ) ∈ ( − ǫ , ǫ ) (cid:0) nd (cid:1) θ + (cid:0) nd − (cid:1) θ + . . . + (cid:0) n (cid:1) θ d (mod Z ) ∈ ( − ǫ , ǫ ) . . . . . . . . . . . . . . . (cid:0) dnd (cid:1) θ + (cid:0) dnd − (cid:1) θ + . . . + (cid:0) dn (cid:1) θ d (mod Z ) ∈ ( − ǫ , ǫ ) , where θ = α d λ . Multiplying (cid:0) ind (cid:1) θ + (cid:0) ind − (cid:1) θ + . . . + (cid:0) in (cid:1) θ d by λ i d ! and summingover i = 1 , . . . , d we get that d X j =1 λ j d ! d X i =0 (cid:0) jnd − i (cid:1) θ i = α d n d (mod Z ) ∈ ( − K d ǫ , K d ǫ ) ⊂ ( − ǫ/d, ǫ/d ) . Choose x i ∈ U i for 1 ≤ i ≤ d . Let x = ( x , x , . . . , x d ) ∈ X × . . . × X d and X bethe orbit x under T = T × T . . . × T d . Then ( X, T ) is a d -step nilsystem. If we let U = ( U × U × . . . × U d ) ∩ X , then we have (7.1).By the property of nilsystems and the discussion above it is easy to see Remark . Let k ∈ N , q i ( x ) be a polynomial of degree d with q i (0) = 0 and ǫ i > ≤ i ≤ k . Then there are a d -step nilsystem ( X, T, µ ) and B ⊂ X with µ ( B ) > k \ i =1 { n ∈ Z : || q i ( n ) || < ǫ i } ⊃ { n ∈ Z : µ ( B ∩ T − n B ∩ . . . ∩ T − dn B ) > } Some preparation. For d ∈ N , define N d = { B ⊆ Z : there are a minimal d -step nilsystem ( X, T ) and an opennon-empty set U of X with B ⊃ { n ∈ Z : U ∩ T − n U ∩ . . . ∩ T − dn U = ∅} . } Hence Theorem C is equivalent to F GP d ⊆ N d . Lemma 7.2. For each d ∈ N , N d is a filter.Proof. Let B , B ∈ N d . To show N d is a filter, it suffices to show B ∩ B ∈ N d . Bydefinition, there exist minimal d -step nilsystems ( X i , T i ) , i = 1 , 2, and a nonemptyopen set U i such that B i ⊃ { n ∈ Z : U i ∩ T − ni U i ∩ . . . ∩ T − dni U i = ∅} . Taking any minimal point x = ( x , x ) ∈ X × X , let X = orb( x, T ), where T = T × T . Note that ( X, T ) is also a minimal d -step nilsystem.Since ( X i , T i ) , i = 1 , 2, are minimal, there are k i ∈ N such that x i ∈ T − k i i U i , i = 1 , 2. Let U = ( T − k U × T − k U ) ∩ X , then U is an open set of X . Note that { n ∈ Z : U ∩ T − n U ∩ . . . ∩ T − dn U = ∅} = \ i =1 , { n ∈ Z : T − k i i U i ∩ T − ( k i + n ) i U i ∩ . . . ∩ T − ( k i + dn ) i U i = ∅} = \ i =1 , { n ∈ Z : U i ∩ T − ni U i ∩ . . . ∩ T − dni U i = ∅} sets Hence B ∩ B ⊃ { n ∈ Z : U ∩ T − n U ∩ . . . ∩ T − dn U = ∅} . That is, B ∩ B ∈ N d and N d is a filter. (cid:3) Definition 7.3. For r ∈ N , define d GP r = { p ( n ) ∈ GP r : { n ∈ Z : p ( n ) (mod Z ) ∈ ( − ǫ, ǫ ) } ∈ N r , ∀ ǫ > } . Remark . It is clear that for p ( n ) ∈ GP r , p ( n ) ∈ d GP r if and only if − p ( n ) ∈ d GP r .Since N r is a filter, if p ( n ) , p ( n ) , · · · , p k ( n ) ∈ d GP r then p ( n ) + p ( n ) + · · · + p k ( n ) ∈ d GP r . Moreover by the definition of d GP r , we know that F GP r ⊂ N r if and only if d GP r = GP r .Since we will use induction to show Theorem C, thus we need to obtain someresults under the following assumption, that is for some d ≥ F GP d − ⊆ N d − . Lemma 7.5. Let p ( n ) , q ( n ) ∈ GP d with p ( n ) ≃ d q ( n ) . Under the assumption (7.4) , p ( n ) ∈ d GP d if and only if q ( n ) ∈ d GP d .Proof. It follows from Lemma 6.3, N d being a filter and F GP d − ⊆ N d − ⊆ N d . (cid:3) Theorem 7.6. Under the assumption (7.4) , the following properties are equivalent: (1) F GP d ⊆ N d . (2) P ( n ; α , α , · · · , α d ) ∈ d GP d for any α , α , · · · , α d ∈ R , that is { n ∈ Z : P ( n ; α , α , · · · , α d ) (mod Z ) ∈ ( − ǫ, ǫ ) } ∈ N d for any α , α , · · · , α d ∈ R and ǫ > . (3) SGP d ⊂ d GP d .Proof. The proof is similar to that of Theorem 6.7. (cid:3) Proofs of Theorem C. Now we prove F GP d ⊆ N d by induction on d . When d = 1, since F GP = F SGP and N d is a filer, it is sufficient to show that: for any p ( n ) = an ∈ SGP and ǫ > 0, we have(7.5) { n ∈ Z : p ( n ) (mod Z ) ∈ ( − ǫ, ǫ ) } ∈ N . This is easy to be verified.Now we assume that for d ≥ F GP d − ⊆ N d − , i.e. (7.4) holds. Then it fol-lows from Theorem 7.6 that under the assumption (7.4), to show F GP d ⊆ N d , it issufficient to show that P ( n ; β , β , . . . , β d ) ∈ d GP d , for any β , β , · · · , β d ∈ R .Fix β , β , . . . , β d ∈ R . We divide the remainder of the proof into two steps. . Huang, S. Shao and X.D. Ye 39 Step 1. We are going to show P ( n ; β , β , · · · , β d ) ≃ d d X ℓ =1 X j , ··· jℓ ∈ N j ··· + jℓ = d ( − ℓ − λU ( n ; j , j , · · · , j ℓ ) , where as in the proof of Theorem B, we define(7.6) U ( n ; j ) = n j j ! j Y r =1 α r , ≤ j ≤ d. And inductively for ℓ = 2 , , · · · , d define U ( n ; j , j , · · · , j ℓ ) = ( U ( n ; j , · · · , j ℓ − ) − ⌈ U ( n ; j , · · · , j ℓ − ) ⌉ ) n j ℓ j ℓ ! j ℓ Y r =1 α P ℓ − t =1 j t + r = ( U ( n ; j , · · · , j ℓ − ) − ⌈ U ( n ; j , · · · , j ℓ − ) ⌉ ) L ( n j ℓ j ℓ ! j ℓ Y r ℓ =1 α P ℓ − t =1 j t + r ℓ )for j , j , · · · , j ℓ ≥ j + · · · + j ℓ ≤ d (see (4.1) for the definition of L ).In fact, let λ , λ , . . . , λ d ∈ Z and λ ∈ N satisfying (7.3). Put α = β /λ, α = β , α = β , . . . , α d = β d . Then P ( n ; β , β , . . . , β d ) = λP ( n ; α , α , . . . , α d ) . Note that in proof of Theorem B we have(7.7) P ( n ; α , α , · · · , α d ) ≃ d d X ℓ =1 X j , ··· jℓ ∈ N j ··· + jℓ = d ( − ℓ − U ( n ; j , j , · · · , j ℓ ) . Since λ is an integer, we have λP ( n ; α , α , · · · , α d ) ≃ d d X ℓ =1 X j , ··· jℓ ∈ N j ··· + jℓ = d ( − ℓ − λU ( n ; j , j , · · · , j ℓ ) . That is, P ( n ; β , β , · · · , β d ) ≃ d d X ℓ =1 X j , ··· jℓ ∈ N j ··· + jℓ = d ( − ℓ − λU ( n ; j , j , · · · , j ℓ ) . Hence, by Lemma 7.5, to show P ( n ; β , β , · · · , β d ) ∈ d GP d , it suffices to show(7.8) d X ℓ =1 X j , ··· jℓ ∈ N j ··· + jℓ = d ( − ℓ − λU ( n ; j , j , · · · , j ℓ ) ∈ d GP d . sets Now choose x = ( x ki ) ≤ k ≤ d, ≤ i ≤ d − k +1 ∈ R d ( d +1) / with x i = α i for i = 1 , , · · · , d and x ki = 0 for 2 ≤ k ≤ d and 1 ≤ i ≤ d − k + 1. Let A = M ( x ) = α . . . α . . . . . . α d − 00 0 0 . . . α d . . . For n ∈ N , if x ( n ) = ( x ki ( n )) ≤ k ≤ d, ≤ i ≤ d − k +1 ∈ R d ( d +1) / satisfies M ( x ( n )) = A n ,then by Lemma 6.1 and Remark 6.2,(7.9) x ki ( n ) = (cid:0) nk (cid:1) α i α i +1 · · · α i + k − for 1 ≤ k ≤ d and 1 ≤ i ≤ d − k + 1.Let X = G d / Γ be endowed with the metric ρ in Lemma 3.7 and T be the nil-rotation induced by A ∈ G d , i.e. B Γ AB Γ for B ∈ G d . Since G d is a d -stepnilpotent Lie group and Γ is a uniform subgroup of G d , ( X, T ) is a d -step nilsystem.Let x = Γ ∈ X and Z be the closure of the orbit orb( x , T ) of e in X . Then ( Z, T )is a minimal d -step nilsystem. We consider ρ as a metric on Z . Step 2. For any ǫ > 0, we are going to show there is a nonempty open set U of Z such that { n ∈ Z : d X ℓ =1 X j , ··· jℓ ∈ N j ··· + jℓ = d ( − ℓ − λU ( n ; j , j , · · · , j ℓ ) (mod Z ) ∈ ( − ǫ, ǫ ) }⊇ { n ∈ Z : U ∩ T − n U ∩ . . . ∩ T − dn U = ∅} . (7.10)That means d P ℓ =1 P j , ··· jℓ ∈ N j ··· + jℓ = d ( − ℓ − λU ( n ; j , j , · · · , j ℓ ) ∈ d GP d .Fix an ǫ > 0. Take ǫ = min n ǫ K ( P d − i =0 d i ) , o , where K = d X m =1 | λ m | (cid:16) d X t =0 m t (cid:17) ,and let ǫ > e ǫ + ǫ + ... + ǫ d − < ǫ . Let U = { z ∈ Z : ρ ( z, x ) < ǫ } = { c Γ ∈ Z : ρ ( c Γ , Γ) < ǫ } . and let S = { n ∈ Z : U ∩ T − n U ∩ . . . ∩ T − dn U = ∅} . Now we show that S ⊆ n n ∈ Z : d X ℓ =1 X j , ··· jℓ ∈ N j ··· + jℓ = d ( − ℓ − λU ( n ; j , j , · · · , j ℓ ) (mod Z ) ∈ ( − ǫ, ǫ ) o Let n ∈ S . Then U ∩ T − n U ∩ . . . ∩ T − dn U = ∅ . Hence there is some B ∈ G d with B Γ ∈ U ∩ T − n U ∩ . . . ∩ T − dn U. . Huang, S. Shao and X.D. Ye 41 Thus ρ ( A mn B Γ , Γ) < ǫ , m = 0 , , , . . . , d − . Since ρ is right translation invariant, we may assume that ρ ( B, I ) < ǫ , where I isthe ( d + 1) × ( d + 1) identity matrix.For each m ∈ { , , . . . , d } , since ρ ( A mn B Γ , Γ) < ǫ there is some C m ∈ Γ suchthat(7.11) ρ ( A mn BC m , I ) < ǫ . Let A mn BC m = M ( z ( m )), where z ( m ) = ( z ki ( m )) ≤ k ≤ d, ≤ i ≤ d − k +1 ∈ R d ( d +1) / . ThenFrom (7.11), we have || A mn BC m − I || < ǫ by Lemma 3.6, thus | z ki ( m ) | < ǫ , ≤ k ≤ d, ≤ i ≤ d − k + 1 . On the one hand, since | z d ( m ) | < ǫ , we have(7.12) d X m =1 λ m z d ( m ) ∈ ( − Kǫ , Kǫ ) . On the other hand, we have d X m =1 λ m z d ( m ) ≈ (cid:16) d X l =1 ( − l − X j ,j ,...,j l ∈ N j + j + ... + j l = d λU ( n ; j , j , . . . , j l ) (cid:17) + △ (cid:16) ( d + d + . . . + d d − )(2 Kǫ ) (cid:17) . (7.13)Note that for a, b ∈ R and δ > a ≈ b + △ ( δ ) means that a − b (mod Z ) ∈ ( − δ, δ ).Since the proof of (7.13) is long, we put it after Theorem C. Now we continue theproof. By (7.13) and (7.12), we have d X l =1 ( − l − X j ,j ,...,j l ∈ N j + j + ... + j l = d λU ( n ; j , j , . . . , j l ) (mod Z ) ∈ (cid:16) − M (2 Kǫ ) , M (2 Kǫ ) (cid:17) ⊆ ( − ǫ, ǫ ) , where M = 1 + d + . . . + d d − . This means that n ∈ n q ∈ Z : d X l =1 X j ,j ,...,j l ∈ N j + j + ... + j l = d ( − l − λU ( q ; j , j , . . . , j l ) (mod Z ) ∈ ( − ǫ, ǫ ) o . Hence S ⊆ n q ∈ Z : d X l =1 X j ,j ,...,j l ∈ N j + j + ... + j l = d ( − l − λU ( q ; j , j , . . . , j l ) (mod Z ) ∈ ( − ǫ, ǫ ) o . Thus we have proved (7.10) which means d P ℓ =1 P j , ··· jℓ ∈ N j ··· + jℓ = d ( − ℓ − λU ( n ; j , j , · · · , j ℓ ) ∈ d GP d . The proof of Theorem C is now finished. sets Proof of (7.13). Since ρ ( B, I ) < ǫ , by Lemma 3.6,(7.14) || B − I || < e ǫ + ǫ + ... + ǫ d − < ǫ < / . Denote B = M ( y ), where y = ( y ki ) ≤ k ≤ d, ≤ i ≤ d − k +1 ∈ R d ( d +1) / . From (7.14), | y ki | < ǫ , ≤ k ≤ d, ≤ i ≤ d − k + 1 . For m = 1 , , · · · , m , Recall that C m ∈ Γ satisfies(7.15) ρ ( A mn BC m , I ) < ǫ . Denote C m = M ( h ( m )), where h ( m ) = ( − h ki ( m )) ≤ k ≤ d, ≤ i ≤ d − k +1 ∈ Z d ( d +1) / . From(7.15), we have || A mn BC m − I || < ǫ , m = 1 , , . . . , d. Let A mn B = M ( w ( m )), where w ( m ) = ( w ki ( m )) ≤ k ≤ d, ≤ i ≤ d − k +1 ∈ R d ( d +1) / . Then w ki ( m ) = x ki ( mn ) + (cid:16) k − X j =1 x ji ( mn ) y k − ji + j (cid:17) + y ki = (cid:0) mnk (cid:1) α i α i +1 · · · α i + k − + k − X j =1 (cid:0) mnj (cid:1) α i α i +1 · · · α i + j − y k − ji + j + y ki , ( mn ) k k ! α i . . . α i + k − + k − X j =1 m j a ki ( j ) + a ki (0) , (7.16)where m = 1 , , . . . , d , a ki ( j ) does not depend on m and | a ki (0) | = | y ki | < ǫ .Recall that z ( m ) = ( z ki ( m )) ≤ k ≤ d, ≤ i ≤ d − k +1 ∈ R d ( d +1) / satisfies A mn BC m = M ( z ( m )). Hence(7.17) z ki ( m ) = w ki ( m ) − (cid:16) k − X j =1 w ji ( m ) h k − ji + j ( m ) (cid:17) − h ki ( m ) . From || A mn BC m − I || < ǫ , we have | z ki ( m ) | < ǫ , ≤ k ≤ d, ≤ i ≤ d − k + 1 . Note that h ki ( m ) ∈ Z , and we have h ki ( m ) = l w ki ( m ) − k − X j =1 w ji ( m ) h k − ji + j ( m ) m . Let u ki ( m ) = w ki ( m ) − k − X j =1 w ji ( m ) h k − ji + j ( m ) . Then | u ki ( m ) − h ki ( m ) | = | z ki ( m ) | < ǫ < / . Recall that for a, b ∈ R and δ > a ≈ b + △ ( δ ) means a − b (mod Z ) ∈ ( − δ, δ ). . Huang, S. Shao and X.D. Ye 43 Claim: Let ≤ r ≤ d − and v r (0) , v r (1) , . . . , v r ( r ) ∈ R . Then for each ≤ r ≤ d − r − and ≤ j ≤ r + r , there exist v r,r ( j ) ∈ R such that (1) d X m =1 λ m (cid:16) r X t =0 m t v r ( t ) (cid:17) h d − r r ( m ) ≈ λ ( v r ( r ) − ⌈ v r ( r ) ⌉ ) n d − r ( d − r )! α r . . . α d − d − r − X r =1 d X m =1 λ m (cid:16) r + r X t =1 m t v r,r ( t ) (cid:17) h d − r − r r + r ( m )+ △ (2 Kǫ )(2) v r,r ( r + r ) = (cid:16) v r − ⌈ v r ⌉ (cid:17) n r r ! α r +1 . . . α r + r for all ≤ r ≤ d − r − .Proof of Claim. Since | u d − r r ( m ) − h d − r r ( m ) | < ǫ , we have (cid:12)(cid:12)(cid:12) d X m =1 λ m (cid:16) r X t =0 m t ( v r ( r ) − ⌈ v r ( r ) ⌉ ) (cid:17)(cid:12)(cid:12)(cid:12) ≤ d X m =1 | λ m | (cid:16) r X t =0 m t (cid:17) ≤ (cid:16) d X m =1 | λ m | (cid:17)(cid:16) r X t =0 m t (cid:17) = K. Hence d X m =1 λ m (cid:16) r X t =0 m t v r ( t ) (cid:17) h d − r r ( m ) ≈ d X m =1 λ m (cid:16) r X t =0 m t ( v r ( t ) − ⌈ v r ( t ) ⌉ ) (cid:17) h d − r r ( m ) ≈ d X m =1 λ m (cid:16) r X t =0 m t ( v r ( t ) − ⌈ v r ( t ) ⌉ ) (cid:17) u d − r r ( m )+ △ ( Kǫ ) . (7.18)Then we have d X m =1 λ m (cid:16) r X t =0 m t ( v r ( t ) − ⌈ v r ( t ) ⌉ ) (cid:17) u d − r r ( m ) d X m =1 λ m (cid:16) r X t =0 m t ( v r ( t ) − ⌈ v r ( t ) ⌉ ) (cid:17)(cid:16) w d − r r ( m ) − d − r − X r =1 w r r ( m ) h d − r − r r + r ( m ) (cid:17) . sets From (7.16) we have d X m =1 λ m (cid:16) r X t =0 m t ( v r ( t ) − ⌈ v r ( t ) ⌉ ) (cid:17) w d − r r ( m )= d X m =1 λ m (cid:16) r X t =0 m t ( v r ( t ) − ⌈ v r ( t ) ⌉ ) (cid:17)(cid:16) ( mn ) d − r ( d − r )! α r . . . α d + d − r − X j =0 m j a d − r r ( j ) (cid:17) = d X m =1 λ m m d n d − r ( d − r )! α r . . . α d (cid:16) v r ( t ) − ⌈ v r ( t ) ⌉ (cid:17) + d − X h =1 d X m =1 λ m m h X ≤ t ≤ r ≤ j ≤ d − r − t + j = h ( v r ( t ) − ⌈ v r ( t ) ⌉ ) a d − r r ( j ) ! + d X m =1 λ m (cid:16) v r (0) − ⌈ v r (0) ⌉ (cid:17) a d − r r (0) , and so P dm =1 λ m (cid:16) P rt =0 m t ( v r ( t ) − ⌈ v r ( t ) ⌉ ) (cid:17) w d − r r ( m ) is equal to λ n d − r ( d − r )! α r . . . α d ( v r ( t ) − ⌈ v r ( t ) ⌉ ) + ( d X m =1 λ m )( v r (0) − ⌈ v r (0) ⌉ ) y d − r r ≈ λ n d − r ( d − r )! α r . . . α d ( v r ( t ) − ⌈ v r ( t ) ⌉ )+ △ ( Kǫ ) . The last equation follows from (cid:12)(cid:12)(cid:12) ( d X m =1 λ m )( v r (0) − ⌈ v r (0) ⌉ ) y d − r r (cid:12)(cid:12)(cid:12) ≤ d X m =1 | λ m | ǫ < Kǫ . Then for 1 ≤ r ≤ d − r − 1, by (7.16), we have d X m =1 λ m (cid:16) r X t =0 m t ( v r ( t ) − ⌈ v r ( t ) ⌉ ) (cid:17) w r r ( m ) h d − r − r r + r ( m )= d X m =1 λ m (cid:16) r X t =0 m t ( v r ( t ) − ⌈ v r ( t ) ⌉ ) (cid:17)(cid:16) ( mn ) r ( r )! α r . . . α r + r + r − X j =0 m j a r r ( j ) (cid:17) h d − r − r r + r ( m )= d X m =1 λ m (cid:16) m r + r n r ( r )! α r . . . α r + r ( v r ( t ) − ⌈ v r ( t ) ⌉ )+ r + r − X h =0 m h (cid:0) X ≤ t ≤ r ≤ j ≤ r − t + j = h (cid:0) v r ( t ) − ⌈ v r ( t ) ⌉ (cid:1) a r r ( j ) (cid:1)(cid:17) h d − r − r r + r ( m ) . . Huang, S. Shao and X.D. Ye 45 Let v r,r ( h ) = X ≤ t ≤ r ≤ j ≤ r − t + j = h (cid:0) v r ( t ) − ⌈ v r ( t ) ⌉ (cid:1) a r r ( j ) , ≤ h ≤ r + r − v r,r ( r + r ) = ( n ) r ( r )! α r . . . α r + r ( v r ( t ) − ⌈ v r ( t ) ⌉ ) ; v r,r (0) = ( v r (0) − ⌈ v r (0) ⌉ ) a r r (0) = ( v r (0) − ⌈ v r (0) ⌉ ) y r r (0) . It is easy to see that | v r,r (0) | < ǫ . Then d X m =1 λ m (cid:16) r X t =0 m t ( v r ( t ) − ⌈ v r ( t ) ⌉ ) (cid:17) w r r ( m ) h d − r − r r + r ( m )= d X m =1 λ m (cid:16) r + r X t =0 m t v r,r ( t ) (cid:17) h d − r − r r + r ( m ) . To sum up, we have d X m =1 λ m (cid:16) r X t =0 m t ( v r ( t ) − ⌈ v r ( t ) ⌉ ) (cid:17) u d − r r ( m ) ≈ λ ( n ) d − r ( d − r )! α r . . . α d ( v r ( t ) − ⌈ v r ( t ) ⌉ ) − d − r − X r =1 d X m =1 λ m (cid:16) r + r X t =0 m t v r,r ( t ) (cid:17) h d − ( r + r )1+ r + r ( m ) ! + △ ( Kǫ ) . Together with ( 7.18), we have d X m =1 λ m (cid:16) r X t =0 m t v r ( t ) (cid:17) h d − r r ( m ) ≈ λ ( n ) d − r ( d − r )! α r . . . α d (cid:16) v r ( t ) − ⌈ v r ( t ) ⌉ (cid:17) − d − r − X r =1 d X m =1 λ m (cid:16) r + r X t =0 m t v r,r ( t ) (cid:17) h d − ( r + r )1+ r + r ( m ) ! + △ (2 Kǫ ) . The proof of the claim is completed. (cid:3) We will use the claim repeatedly. First using (7.17) we have d X m =1 λ m z d ( m ) ≈ d X m =1 λ m (cid:18) w d ( m ) − d − X j =1 w j ( m ) h d − j j ( m ) (cid:19) . By (7.16), we have d X m =1 λ m w d ( m ) ≈ d X m =1 λ m m d n d d ! α . . . α d + d X m =1 λ m y d ≈ λ n d d ! α . . . α d + △ ( Kǫ ) . sets Using this, (7.16) and the claim, we have d X m =1 λ m z d ( m ) ≈ λ n d d ! α . . . α d − d X m =1 λ m d − X j =1 (cid:16) m j n j j ! α . . . α j + j − X t =0 m t a j ( t ) (cid:17) h d − j j ( m )+ △ ( Kǫ ) ≈ λ n d d ! α . . . α d − d − X j =1 λ n d − j ( d − j )! α j . . . α d (cid:16) n j j ! α . . . α j − ⌈ n j j ! α . . . α j ⌉ (cid:17) + d − X j =1 d − j − X j =1 d X m =1 λ m (cid:18) m j + j (cid:16) n j j ! α . . . α j − ⌈ n j j ! α . . . α j ⌉ (cid:17) n j j ! α j . . . α j + j + j + j − X t =0 m t v j ,j ( t ) (cid:19) h d − ( j + j )1+ j + j ( m ) ! + △ ( (cid:0) d − K + K (cid:1) ǫ ) . Note that here we use v j ( t ) = a j ( t ) , t = 0 , , . . . , j − v j ( j ) = n j j ! α . . . α j .Recall the definition of U ( · ): n d d ! α . . . α d = U ( n ; d ) , (cid:16) n j j ! α . . . α j − ⌈ n j j ! α . . . α j ⌉ (cid:17) n j j ! α j . . . α j + j = U ( n ; j , j ) . Substituting these in the above equation, we have d X m =1 λ m z d ( m ) ≈ λU ( n ; d ) − d − X j =1 λU ( n ; j , d − j )+ d − X j =1 d − j − X j =1 d X m =1 λ m (cid:18) m j + j U ( n ; j , j ) + j + j − X t =0 m t v j ,j ( t ) (cid:19) h d − ( j + j )1+ j + j ( m ) ! + △ (2 dKǫ ) Using the claim again, we have: d X m =1 λ m z d ( m ) ≈ λU ( n ; d ) − d − X j =1 λU ( n ; j , j ) + d − X j =1 d − j − X j =1 λU ( n ; j , j , d − j − j ) − d − X j =1 d − j − X j =1 d − ( j + j ) − X j =1 d X m =1 λ m (cid:18) m j + j + j U ( n ; j , j , j )+ j + j + j − X t =0 m t v j ,j ,j ( t ) (cid:19) h d − ( j + j + j )1+ j + j + j ( m ) ! + △ (2 dKǫ + 2 d Kǫ ) . Inductively, we have P dm =1 λ m z d ( m ) ≈ (cid:16) d X l =1 ( − l − X j ,j ,...,j l ∈ N j + j + ... + j l = d λU ( n ; j , j , . . . , j l ) (cid:17) + △ (2 dKǫ + 2 d Kǫ + . . . + 2 d d − Kǫ ) ≈ (cid:16) d X l =1 ( − l − X j ,j ,...,j l ∈ N j + j + ... + j l = d λU ( n ; j , j , . . . , j l ) (cid:17) + △ (cid:16) ( d + d + . . . + d d − )(2 Kǫ ) (cid:17) . The proof of (7.13) is now finished. . Huang, S. Shao and X.D. Ye 47 Applications Our main results can be applied to get results in the theory of dynamical systems.As the limitation of the length of the paper, here we only state the results and thedetailed proofs will appear in a forthcoming paper by the same authors.8.1. d -step almost automorpy. The notion of almost automorphy was first intro-duced by Bochner in 1955 in a work of differential geometry [7, 8]. Veech showedthat each almost automorphic minimal system is an almost one-to-one extensionof a compact metric abelian group rotation [28]. Let ( X, T ) be a minimal systemand d ∈ N . ( X, T ) is called a d -step almost automorphic system if it is an almostone-to-one extension of a d -step nilsystem. Let π : X −→ Y be the almost one-to-one extension with Y being a d -step nilsystem. A point x ∈ X is d -step almostautomorphic if π − π ( x ) = { x } .Using the main results of this paper, we show that F P oi d , F Bir d and F d, can beused to characterize d -step almost automorphy, i.e. in some sense, F P oi d and F ∗ d, can not be distinguished “dynamically”. Similar results can be found in the nextsubsections. Theorem 8.1. [22] Let ( X, T ) be a minimal system, x ∈ X and d ∈ N . Then thefollowing statements are equivalent (1) x is d -step almost automorphic. (2) N ( x, V ) ∈ F ∗ P oi d for each neighborhood V of x . (3) N ( x, V ) ∈ F ∗ Boi d for each neighborhood V of x . (4) N ( x, V ) ∈ F d, for each neighborhood V of x . Regionally proximal relation of order d . Regionally proximal relation. Regionally proximal relation plays a very im-port role in the theory of topological dynamics. It is the main tool to characterize theequicontinuous structure relation S eq ( X ) of a system ( X, T ); i.e. to find the small-est closed invariant equivalence relation R ( X ) on ( X, T ) such that ( X/R ( X ) , T ) isequicontinuous. Veech [29] gave the first proof of the fact that the regionally proxi-mal relation is an equivalence one. Also he showed that Poincar´e sets can be usedto characterize regionally proximal relation.8.2.2. Regionally proximal relation of order d . In [17] Host and Kra defined a d -stepnilfactor for each ergodic system, see also [31]. To get a similar factor in topologicaldynamics Host, Maass and Kra introduced the notion of regionally proximal relationof higher order. Definition 8.2. [20, 19] Let ( X, T ) be a system and let d ≥ x, y ) ∈ X × X is said to be regionally proximal of order d if for any δ > x ′ , y ′ ∈ X and a vector n = ( n , . . . , n d ) ∈ Z d and ǫ ∈ { , } d such that d ( x, x ′ ) < δ, d ( y, y ′ ) < δ , and ρ ( T n · ǫ x ′ , T n · ǫ y ′ ) < δ for any ǫ = (0 , , . . . , , where n · ǫ = n ǫ + . . . + n d ǫ d . sets The set of regionally proximal pairs of order d is denoted by RP [ d ] ( X ), which iscalled the regionally proximal relation of order d .It is easy to see that RP [ d ] ( X ) is a closed and invariant relation for all d ∈ N .When d = 1, RP [ d ] ( X ) is nothing but the classical regionally proximal relation. In[19], for distal minimal systems the authors showed that RP [ d ] ( X ) is a closed invari-ant equivalence relation, and the quotient of X under this relation is its maximal d -step nilfactor. Recently, these results are shown to be true for general minimalsystems by Shao-Ye [27]. We remark that a point is d -step almost automorphic ifand only if RP [ d ] [ x ] = { x } . Moreover, we have the following theorem whose proofsare based on the results built in this paper. Theorem 8.3. [22] Let ( X, T ) be a minimal system and d ∈ N . The followingstatements are equivalent: (1) ( x, y ) ∈ RP [ d ] ( X )(2) N ( x, U ) ∈ F P oi d for each neighborhood U of y . (3) N ( x, U ) ∈ F Bir d for each neighborhood U of y . (4) N ( x, U ) ∈ F ∗ d, for each neighborhood U of y . We remark that for d = 1 the above theorem is known, see for example [29, 9, 21].8.3. Nil d Bohr sets and sets SG d ( P ) . In this article to study Nil d Bohr sets,we use generalized polynomials. In [18], Host and Kra introduced an interesting set( SG d set) to study the Nil d Bohr sets. Here we state some results and give somequestions. First we recall definitions introduced by Host and Kra in [18].8.3.1. Sets SG d ( P ) and SG ∗ d . Let d ≥ P = { p i } i be a (finiteor infinite) sequence in N . The set of sums with gaps of length less than d of P isthe set SG d ( P ) of all integers of the form ǫ p + ǫ p + . . . + ǫ n p n where n ≥ ǫ i ∈ { , } for 1 ≤ i ≤ n , the ǫ i are not all equal to 0,and the blocks of consecutive 0’s between two 1 have length less than d . A subset A ⊆ N is an SG ∗ d -set if A ∩ SG d ( P ) = ∅ for every infinite sequence P in N .Note that in this definition, P is a sequence and not a subset of N . For example,if P = { p , p , . . . } , then SG ( P ) is the set of all sums p m + p m +1 + . . . + p n ofconsecutive elements of P , and thus it coincides with the set ∆( S ) where S = { p , p + p , p + p + p , . . . } . Therefore SG ∗ -sets are the same as ∆ ∗ -sets.In [18] a notion called strongly piecewise- F , written PW- F was introduced andthe following proposition was proved Proposition 8.4. Every SG ∗ d -set is a PW-Nil d Bohr -set. Host and Kra asked the following question. Question 8.5. Is every Nil d Bohr -set an SG ∗ d -set? Though we can not answer this question, we show that they can not be distin-guished dynamically (see Theorems 8.3 and 8.7). Using Theorem B, Question 8.5can be reformulated in the following way: . Huang, S. Shao and X.D. Ye 49 Question 8.6. Let d ∈ N and S be an SG d -set. Is it true that for any k ∈ N , any P , . . . , P k ∈ F SGP d and any ǫ i > , there is n ∈ S such that P i ( n )( mod Z ) ∈ ( − ǫ i , ǫ i ) for all i = 1 , . . . , k ? We remark that since a d -step nilsystem is distal, the above question has anaffirmative answer for any IP-set.8.3.2. SG d sets and regionally proximal relation of order d . We can use SG d sets tocharacterize regionally proximal relation of order d . For each d ∈ N , denote by SG d the collection of all sets SG d ( P ), and F SG d the family generated by SG d . Theorem 8.7. [22] Let ( X, T ) be a minimal system. Then for any d ∈ N , ( x, y ) ∈ RP [ d ] if and only if N ( x, U ) ∈ F SG d for each neighborhood U of y . A direct corollary of Theorem 8.7 is: let ( X, T ) be a minimal system, x ∈ X , and d ∈ N . If x is F ∗ SG d -recurrent, then it is d -step almost automorphic. Since F SG d doesnot have the Ramsey property [22], we do not know if the converse of the abovecorollary holds. References [1] E. Akin, Recurrence in topological dynamics, Furstenberg families and Ellis actions , PlenumPress, New York and London, 1997.[2] L. Auslander, L. Green and F. Hahn, Flows on Homogeneous Spaces , (Annals of MathematicsStudies, 53). Princeton University Press, Princeton, NJ, 1963.[3] V. Bergelson, H. Furstenberg and B. Weiss, Piecewise-Bohr Sets of Integers and Combi-natorial Number Theory , (Algorithms and Combinatorics, 26). Springer, Berlin, 2006, pp.13-37.[4] V. Bergelson, B. Host and B. Kra, Multiple recurrence and nilsequences. With an appendixby Imre Ruzsa , Invent. Math., (2005), no. 2, 261–303.[5] V. Bergelson and A. Leibman, Distribution of values of bounded generalized polynomials ,Acta Math., (2007), no. 2, 155-230.[6] V. Bergelson and R. McCutcheon, An ergodic IP polynomial Szemerdi theorem. Mem. Amer.Math. Soc. , (2000), no. 695.[7] S. Bochner, Curvature and Betti numbers in real and complex vector bundles , Universit´a ePlolitecnico di Toorino, Rendiconti del seminario matematico, vol. 15 (1955-56), 225–254.[8] S. Bochner, A new approach to almost periodicity , Proc. Nat. Acad. Sci. USA, (1962),2039–2043.[9] M. Boshernitzan and E. Glasner, On two recurrence problems , Fund. Math., (2009),113–130.[10] L. J. Corwin and F. P. Greenleaf, Representations of nilpotent Lie groups and their applica-tions . Part I. Basic theory and examples, Cambridge Studies in Advanced Mathematics 18,Cambridge University Press, Cambridge, 1990. viii+269 pp.[11] N. Frantzikinakis, E. Lesigne and M. Wierdl, Sets of k -recurrence but not ( k + 1) -recurrence ,Ann. Inst. Fourier (Grenoble), (2006), no. 4, 839–849.[12] H. Furstenberg, Poincar ´ e recurrence and number theory , Bull. Amer. Math. Soc. (N.S.), (1981), no. 3, 211–234.[13] H. Furstenberg, Recurrence in ergodic theory and combinatorial number theory , M. B. PorterLectures. Princeton University Press, Princeton, N.J., 1981. sets [14] H. Furstenberg and Y. Katznelson, An ergodic Szemerdi theorem for IP-systems and combi-natorial theory , J. Analyse Math., (1985), 117-168.[15] B. Green and T. Tao, The quantitative behaviour of polynomial orbits on nilmanifolds , toappear in Annals of Math.[16] B. Green, T. Tao and T. Ziegler, An inverse theorem for the Gowers U s +1 [ N ] norm ,arXiv:1009.3998.[17] B. Host and B. Kra, Nonconventional averages and nilmanifolds , Ann. of Math., (2005)398-488.[18] B. Host and B. Kra, Nil-Bohr sets of integers , Ergod. Th. & Dynam. Sys., (2011), no. 1,113-142.[19] B. Host, B. Kra and A. Maass, Nilsequences and a Structure Theory for Topological Dynam-ical Systems , Advances in Mathematics, 224 (2010) 103–129.[20] B. Host and A. Maass, Nilsyst`emes d’ordre deux et parall´el´epip`edes , Bull. Soc. Math. France, (2007) 367-405.[21] W. Huang, P. Lu and X. Ye, Measure-theoretical sensitivity and equicontinuity , Israel J. ofMath., (2011), 233-284.[22] W. Huang, S. Shao and X. Ye, Higher order almost automorphy, recurrence sets and region-ally proximal relation , in preparation.[23] Y. Katznelson, Chromatic numbers of Cayley graphs on Z and recurrence , Paul Erd´os andhis mathematics (Budapest, 1999), Combinatorics, (2001), 211-219.[24] I. Kˇr´ıˇz, Large independent sets in shift-invariant graphs. Solution of Bergelson’s problem ,Graphs Combin., (1987), 145-158.[25] A. Leibman, Pointwise convergence of ergodic averages for polynomial sequences of transla-tions on a nilmanifold , Ergod. Th. & Dynam. Sys., (2005), 113-201.[26] A.I. Malcev, On a class of homogeneous spaces , Amer. Math. Soc. Translation, no. 39. (1951).[27] S. Shao and X. Ye, Regionally proximal relation of order d is an equivalence one for minimalsystems and a combinatorial consequence , preprint, arXiv:1007.0189.[28] W. A. Veech, Almost automorphic functions on groups , Amer. J. Math., (1965), 719-751.[29] W. A. Veech, The equicontinuous structure relation for minimal Abelian transformationgroups , Amer. J. Math., (1968), 723-732.[30] B. Weiss, Single Orbit Dynamics , CBMS Reg. Conf. Ser. Math. 95, Amer. Math. Soc.,Providence, RI, 2000.[31] T. Ziegler, Universal characteristic factors and Furstenberg averages . J. Amer. Math. Soc., (2007), no. 1, 53-97. Department of Mathematics, University of Science and Technology of China,Hefei, Anhui, 230026, P.R. China. E-mail address : [email protected] E-mail address : [email protected] E-mail address ::