Distinct-symmetry spin liquid states and phase diagram of Kitaev-Hubbard model
aa r X i v : . [ c ond - m a t . s t r- e l ] A ug Symmetry distinct spin liquid states and phase diagram of Kitaev-Hubbard model
Long Liang , Ziqiang Wang , and Yue Yu , State Key Laboratory of Theoretical Physics, Institute of Theoretical Physics,Chinese Academy of Sciences, P.O. Box 2735, Beijing 100190, China Department of Physics, Boston College, Chestnut Hill, Massachusetts 02467, USA Department of Physics, Center for Field Theory and Particle Physics,and State Key Laboratory of Surface Physics, Fudan University, Shanghai 200433, China (Dated: July 17, 2018)We report the finding of a series of symmetry distinct spin liquid (SL) states and a rich phasediagram in a half-filled honeycomb lattice Hubbard model with spin-dependent hopping amplitude t ′ . We first study the magnetic instability of the system and find two antiferromagnetic (AF) ordersbeyond a critical Hubbard U which increases with the ratio t ′ /t . For t ′ approaching to t , thesemimetal (SM) transforms to a U (1) SL and then to the Kitaev Z SL as U increases. In a widemiddle range of t ′ /t , the latter is replaced by a U (1) SL to SU (2) SL transition. The physicalproperties of the stable SL phases are discussed. I. INTRODUCTION
Searching for the spin liquid (SL) states in stronglycorrelated systems has been one of the most intriguingand important fields . Experimentally, possible SL stateshave been observed recently in quantum frustrated spinsystems . Theoretically, ample numerical evidence ofSL ground state has been found in models of frustratedspin systems and various exactly solvable modelshave been constructed to support the existence of theSL ground state .While most of the studies focused on quantum spinsystems, possible SL states in the Hubbard model nearthe metal-insulator transition have attracted great inter-ests recently. Quantum Monte Carlo studies have pro-duced controversial results amid an active debateover other investigations . Whether a SL groundstate exists remains inconclusive due to the intricate in-terplay of charge and spin dynamics in the quantum crit-ical region of the metal insulator transition and the in-cipient antiferromagnetic (AF) order.We here study the ground state and the phase dia-gram of a generalized t − t ′ − U Hubbard model where t ′ describes nearest neighbor spin-dependent hopping onthe half-filled honeycomb lattice. This model, which wecall a Kitaev-Hubbard model, was introduced by Duanet al, and can be realized in cold atom systems ,has been studied numerically at quarter filling andhalf filling . Our motivation is that as a function of t ′ /t , this model interpolates between the usual Hubbardmodel at t ′ /t = 0 and 1 where the Kitaev Z SL becomes the known ground state in the large on-site U limit . With the latter severing as a reference SLstate, we obtain a rich phase diagram on the U/t − t ′ /t plane that reveals the phase structure of several symme-try distinct SL states on the honeycomb lattice as wellas their competition with several forms of AF order.A SL is a Mott insulator without spin order that hasexotic charge-neutral excitations called spinons which arecoupled by emerging gauge fields . Most commonlystudied SLs are U (1), SU (2), and Z SL - named af- ter the symmetry of the gauge fields, respectively. Inthis work, we show that these SL states emerge on thephase diagram of the Kitaev-Hubbard model at half fill-ing and have yet to be explored by numerical studies .Specifically, we apply the SU (2) slave rotor theory tothe Kitaev-Hubbard model, which recovers the semimetal(SM) phase with stable Dirac points at weak interac-tions, and search for all possible SL phases with gaplessfermionic spinon excitations coupled to the correspond-ing gauge groups. We also calculate the spin suscepti-bility using the random phase approximation (RPA) anddetermine the stability of the SLs against magnetic order-ing on the honeycomb lattice. The physical characters ofthe different SLs that are experimentally distinguishablewill be discussed.The present paper is organized as follows. In sectionII we describe the model and its symmetries. We studythe magnetic instabilities using RPA in section III. Insection IV, we use SU (2) slave rotor theory to study theSL phases. Then we describe how to combine the resultsof RPA and slave rotor to obtain the global phase dia-gram in section V. Finally, we discuss how to measurethe different phases in cold atom experiments in sectionVI. II. THE MODEL AND ITS SYMMETRIES
The Hamiltonian for the Kitaev-Hubbard model on ahoneycomb lattice is given by H = − t X h ij i c † i c j − t ′ X h ij i a c † i σ a c j + U X i n i ↑ n i ↓ , (1)where c i = [ c i ↑ , c i ↓ ] T and c iσ ( c † iσ ) annihilates (creates)an electron with spin σ = ( ↑ , ↓ ) on site i . The t -termis the conventional hopping integral, while the t ′ -termdescribes link-and-spin-dependent hopping, and U is theon-site repulsion. σ a ( a = x, y, z ) are the Pauli matrices. h ij i a denotes the nearest neighbor pairs in the a -link di-rection(see Fig.1). FIG. 1: Honeycomb lattice. a and a are primitive trans-lation vectors. Nearest-neighbor links are divided into threetypes, called x -links, y -links and z -links. We group the electron operators into 2 × Ψ A = " c A ↑ c † A ↓ c A ↓ − c † A ↑ , Ψ B = " c B ↑ − c † B ↓ c B ↓ c † B ↑ , (2)where A and B label sites on the two sub-lattices, anddefine spin and pseudo-spin operators as S a = 14 Tr(Ψ † σ a Ψ) , T a = 14 Tr(Ψ σ a Ψ † ) S a acts on the subspace with odd electron number while T a acts on the subspace with even electron number sothey are commutative with one another.To reveal the symmetry it’s convenient to rewrite theHamiltonian in terms of Ψ A and Ψ B : H = − t X h A,B i Tr(Ψ † A Ψ B ) − t ′ X h A,B i a Tr( σ z Ψ † A σ a Ψ B )+ 2 U X i T i . (3)The t -term clearly preserves spin and pseudo-spin rota-tional symmetries while the t ′ -term also preserves spinand pseudo-spin rotational symmetries if we performsuitable local rotation of Ψs. (See Appendix A.) How-ever, when both t and t ′ are non-zero, both SU (2)symmetries are broken since the symmetry operationsof t -term and t ′ -term are not compatible with eachother. There is an important discrete chiral symme-try associated with the operator: S = P · T where P = e iπT y is particle-hole and T = e iπS y K the time-reversal operation( K is complex conjugation). Becauseof this symmetry, TRS is enforced at half-filling in thelarge U limit as seen in the phase diagram in Fig.5. In contrast to the usual lattice translation T x and T y and theinversion R symmetries that are preserved, the six-foldrotational symmetry is broken due to the t ′ -term. How-ever, the system is invariant under combined π/ A,B → e − i π σ x e − i π σ y Ψ A,B
In the noninteracting limit ( U = 0) , the electrondispersion ε ( k ) is given by ε ( k ) = t | f | + 3 t ′ ± [( t | f | + 3 t ′ ) − | t ′ ( g + g − + 1) − t f | ] / , (4)where g ± = e ik ± ie ik and f ( k ) = 1 + e ik + e ik . k = k · a = k x / √ k y , k = k · a = − k x / √ k y ,which maintains the original Dirac points on the honey-comb lattice at ( k x , k y ) = ( ± π/ ,
0) where g + g − = − f = 0. When t ′ = 0, the dispersion near the Diracpoint is ε ( k ) = ±√ tk/
2. When t ′ = 0, the low energyphysics is controlled by t ′ and the dispersion near theDirac points becomes ε ( k ) = ± t ′ k/
2. Simple dimensioncounting shows that the Hubbard term is perturbativelyirrelevant in the weak coupling limit.
III. MAGNETIC INSTABILITIES AND AFORDER
In this section we determine the magnetic phaseboundaries by calculating the static spin susceptibility χ ( q ) by RPA. This kind of calculation is standard sowe only present the main results here and the details canbe found in Appendix B.We find that when t ′ /t < .
57, the peak in χ ( q ) islocated at the Γ point which means the magnetic phaseis dominated by N´eel order. For t ′ /t > .
57, the peaksplits into two, indicative of the tendency toward incom-mensurate AF order. Fig.2 (a) shows an example of thedouble-peak structure around the Γ in the susceptibilityin the incommensurate N´eel (i-N´eel) regime. The mag-netic phase diagram is shown in Fig.2 (b) where the SMand the magnetic phase boundary marks the onset of thedivergence in the magnetic susceptibility χ ( q ) . Theappearance of the i-N´eel AF regime has the same originas that of the Kitaev SL phase due to the competitionbetween the N´eel order and the rotated N´eel order .Microscopically, this competition arises from the t - and t ′ -terms and pushes the magnetic phase boundary dra-matically toward large U , enabling the possible stableSL phases in the enlarged nonmagnetic regime in Fig.2when t ′ /t > .
57. In the following, we will focus on the t ′ /t > .
57 regime and use the SU (2) slave rotor theoryto study the emergent SL phases. −4 −2 0 2 40.1650.170.1750.180.185 q x χ t’/t=1 U/t t ’ /t i-N´eelN´eel FIG. 2: (color online) Upper: An example of the double peaksof χ ( q ) in i-N´eel regime. Lower: i-N´eel refers to incommen-surate AF order and there is no magnetic order in the blankarea. IV. SU(2) SLAVE ROTOR THEORYA. A brief review of SU (2) slave rotor theory Here we give a brief review of the SU (2) slave ro-tor theory; more details can be found in . We startfrom Eq.3 and decouple the Hubbard interaction by aHubbard-Stratonovich (HS) transformation i X i ∈ A,B
Tr(Ψ i φ ai σ a Ψ † i ) + 316 U X i ∈ A,B
Tr( φ ai σ a ) where φ a is a three-component HS field. To reveal the SU (2) structure of the theory, we rotate the order pa-rameters in the pseudo-spin space: φ ai σ a → Z † i φ ai σ a Z i where Z i is a time-dependent SU (2) matrix and can beparametrized as: Z i = (cid:20) z i, z i, − z ∗ i, z ∗ i, (cid:21) under the constraint | z i, | + | z i, | = 1. Such a rotationtransforms the electron operator Ψ i to F i = Ψ i Z † i but doesn’t affect the spin operator S . We thus call F i thespinon carrying the spin degrees of freedom and Z i the SU (2) rotor tracking the pseudo-spin(charge) degrees offreedom. With a change of variable φ ai σ a → φ ai σ a + 2 iZ i ∂ τ Z † i the SU (2) action reads S = S t + Z β d τ (cid:26) X i ∈ A,B
Tr( F i ( ∂ τ + i φ ai σ a ) F † i )+ 12 U ′ X i Tr( 12 φ ai σ a + iZ i ∂ τ Z † i ) (5)+ X i iλ i [ 12 Tr( Z † i Z i ) − (cid:27) , with S t = Z β d τ (cid:26) − t X h A,B i Tr( Z B Z † A F † A F B ) − t ′ X h A,B i a Tr( Z B σ z Z † A F † A σ a F B ) (cid:27) (6)where U ′ = 2 U/ λ i is a local Lagrange multiplierimposing the Z ∈ SU (2) constraint .Eq.(5) describes a strongly coupled SU (2) gauge the-ory, where Z and F are matter fields and φ are the tem-poral components of gauge fields. The gauge transforma-tions are: F i → F i W i , Z i → W † i Z i , φ ai σ a → φ ai σ a − iW i ∂ τ W † i The quartic matter fields in the hopping terms in Eq.(6)can be further decoupled using the HS transformation,giving the spatial components of gauge fields. FollowingLee and Lee we decouple the hopping terms: S t = Z β d τ (cid:26) t X h A,B i Tr( η AB η † AB ) − t X h A,B i Tr( η AB Z B Z † A ) − t X h A,B i Tr( η † AB F † A F B ) + t ′ X h A,B i a Tr( η ′ AB η ′† AB ) (7) − t ′ X h A,B i a [Tr( η ′ AB Z B σ z Z † A ) + Tr( η ′† AB F † A σ a F B )] (cid:27) We can write η AB = | η | e iθ AB e i ( c aAB − id aAB ) σ a and η ′ AB = i | η ′ | e iθ ′ AB e i ( c ′ aAB − id ′ aAB ) σ a . The action becomes complexfor fluctuations of θ AB , d AB , φ i and λ i . To get sad-dle point solutions with real free energy we performanalytic continuations, θ AB → i ˜ θ AB , d AB → i ˜ d AB , iφ i → ˜ φ i and iλ i → ˜ λ i , where quantities with tildes arereal. We can obtain η AB = | η Z | e i ( c aAB + ˜ d aAB ) σ a , η † AB = | η F | e − i ( c aAB − ˜ d aAB ) σ a , η ′ AB = i | η ′ Z | e i ( c ′ aAB + ˜ d ′ aAB ) σ a and η ′ † AB = − i | η ′ F | e − i ( c ′ aAB − ˜ d ′ aAB ) σ a . η/η ′ is a real/imaginarynumber times an SU (2) matrix.Since we are interested in the half filling case, we willtake ˜ φ ai = 0 ans¨atz for the temporal components of thegauge fields and relax the constraint by setting λ i = λ ,as is always done in slave particle theories. Despite thelarge gauge fluctuations, it is possible to obtain decon-fined phases by different mean field ans¨atz . B. Z and SU (2) SL phases
The generic ans¨atz, η † AB = a F σ , η ′† AB,a = a ′ F σ a , η AB = a Z σ , η ′ AB,a = a ′ Z σ a , breaks the SU (2) gauge symmetry to Z , where the sub-script a = x, y, z denotes the bond type. It preservesTRS and is not thus valid in the weak coupling. De-noting t ↑ = − ta F f − t ′ a ′ F , t ↓ = − ta F f + t ′ a ′ F , ∆ ↑ = − t ′ a ′ F ( e ik + e ik ), and ∆ ↓ = − t ′ a ′ F ( e ik − e ik ), the dis-persion of the spinon is given by ε F = ± | t σ ± ∆ σ | , (8)which is the same as that obtained in . The spinonband structure in the Z phase is depicted in Fig. 3(Upper panel). mean field equations are derived in Ap-pendix C. There are two critical lines in the mean fieldsolutions. For U < U c , the solution gives an improper p -wave superfluid phase that does not recover the SMwhen U →
0. When
U > U c , rotors are gapped andthe Z SL arises. Remarkably, a second critical line U c exists for t ′ /t . .
91, such that when
U > U c , a ′ F and a ′ Z vanish, i.e., the spin dependent hopping renormalizesto zero. The SU (2) gauge symmetry is thus restored andthe system enters an SU (2) SL phase.It is instructive to study the Z ans¨atz for large- U morecarefully and compare to exact results at t ′ /t = 1. Theeffective chemical potential of rotors is now λ = U ′ =2 U/ Z mean field equations are given by a Z = 16 tN X k ∂ε F,σ ± ∂a F , a ′ Z = 16 t ′ N X k ∂ε F,σ ± ∂a ′ F (9) a F = 124 U ′ tN X k ∂ε Z, ± ∂a Z , a ′ F = 124 U ′ t ′ N X k ∂ε Z, ± ∂a ′ Z , where ε F σ ± and ε Z, ± are the spinon and the rotor dis-persions respectively. ε F σ ± gives Majorana fermion ex-citations at the Dirac point and six gapped flat bands.Note that ε Z, ± ∼ t and then a F , a ′ F ∼ t/U , as expectedfor the charge and spin excitations in this limit. When t = t ′ , the velocity of the linear spinon dispersion in theKitaev model, i.e., in the large U limit, is determined by J = t U . In order to be consistent with this velocity, theparameter a F in the mean field spinon dispersion needs −0.4−0.200.20.4 Γ M K Γ −1−0.500.51 Γ M K Γ FIG. 3: (color online) Upper: Spinon band structure in Z SL phase. U = 4 t and t ′ = t . Lower: Spinon band structurein U (1) SL phase. U = 3 . t and t ′ = 0 . t . Bands shown inblue solid line are occupied, in red dash line are unoccupied.Spinon dispersion in SU (2) phase looks like the one in Z phase with no (nearly) flat band. to be rescaled to a F = J t , which amounts to rescale theHubbard U by a factor α ≈ .
572 at t ′ /t = 1 (see Ap-pendix C). Note that it is well-known that the Hubbard U needs to be rescaled in the mean field approximationof the slave rotor theory . We would like to point outthat the rescale of U only affects the results quantita-tively. We will demonstrate below that α is essentiallyindependent of t ′ /t in the regime where the slave rotortheory can be considered reliable. We solve the meanfield equations self-consistently with the rescaled U . C. U (1) SL phases
To recover the semi-metal phase in small U regime, weconsider the following ans¨atz, η † AB = a F σ , η ′† AB,a = a zF σ z , η AB = a Z σ , η ′ AB,a = a zZ σ z , which breaks the SU (2) gauge symmetry to U (1) in gen-eral. The dispersion of the spinon is thus given by ε F = t a F | f | + 3 t ′ a z F ± (cid:2) ( t a F | f | + 3 t ′ a z F ) − | t ′ a z F ( g + g − + 1) − t a F f | (cid:3) / , (10)which has the same form as Eq. (4) with renormalizedhoppings t → ta F and t ′ → t ′ a zF . The spinon band struc-ture in the U (1) phase is depicted in Fig. 3 (Lower panel).Hence Hence, we expect it to be favored near the weakcoupling where the TRS is broken. At the Dirac points,the linear dispersion ε F = ± t ′ | a zF | k/
2. We again obtaintwo critical lines ˜ U c and ˜ U c > ˜ U c . When U < ˜ U c , therotors condense and the system is in the weak couplingSM phase. For U > ˜ U c , the rotors are gapped and the U (1) SL phase emerges. For U > ˜ U c , both a zF and a zZ vanish and the system enters the SU (2) SL phase. U/t t ’ /t Semi−Metal Z SLSU(2) SL
U(1) SL
FIG. 4: (color online) Phase diagram of the SL states in theabsence of magnetic order. Here we take α = 0 . At a given point in the
U/t - t ′ /t plane, which SL stateis favored can be determined by comparing the mean fieldground state energies among the U (1), Z and SU (2)ans¨atz. The obtained slave-rotor phase diagram is shownin Fig.4.Before ending this section, we discuss the stability ofthe SL states. To determine whether the SLs are stableone have to go beyond the mean field theory and con-sider the gauge fluctuations. We briefly discuss this issuehere. The Z spin liquid is stable because the gauge fluc-tuations are gapped. The SU (2) spin liquid is the onestudy by Hermele , in this phase the low energy effec-tive theory is gapless Dirac fermion coupled to compact SU (2) gauge fields. Large N expansion shows that whenthe number of fermion flavors is large enough, this spinliquid phase is stable . The effective theory of U (1) spinliquid is gapless Dirac fermions coupled to compact U (1)gauge field, this phase may also be stable against instan-ton effect . V. DETERMINATION OF THE GLOBALPHASE DIAGRAM
For the Kitaev-Hubbard model, in particular, a numer-ical calculation based on a variational cluster approxima-tion and cluster perturbation theory showed that the SLphase is unstable against the SM and AF states when t ′ /t is smaller than a certain value . We now explainhow to combine the SL phase diagram in Fig.4 and themagnetic phase diagram in Fig. 2 to arrive at the global phase diagram shown in Fig.5. First, it is known when t ′ /t ∼ t ′ /t . The numerical work in Ref. showed that there isa tricritical point for the SL, SM and AF phases. Thistricritical point in our result corresponds to the discon-tinuity point in the slope of the phase boundary in Fig.2(b). Remarkably, we find that the lower bound of the SLphase touches the singular point of the magnetic phaseboundary, forming the tricritical point observed by nu-merical simulations . We stress that the tricritical pointemerges in our theory without the need to change therescaling factor α determined by the exact solution of theKitaev model at t ′ /t = 1 and lends further support foran essentially t ′ /t -independent α in the SL regime. Weemphasize that the topology of the SL phase diagram isnot affected by varying α (see Appendix C 3). However,the tricritical point exists only if α ≈ . U , the SU (2) SL becomes unstable to the AFi-N´eel phase. However, the AF phase terminates when itmeets the Z SL because the Kitaev SL has lower energy.Finally, we would like to remark that although the gen-eral phase structure of our theory captures that of thenumerical results with unprecedented symmetry distinctSL phases, the phase boundaries between the SM, AFand SL phases as well as the tricritical point only quali-tatively agree with the numerical results in Refs. .The exact determination of the phase boundaries is be-yond the scope of the current work.It’s time to describe our main results shown in thephase diagram shown in Fig.5. Generally speaking, wefound three types of phases: The SM phase for weak cou-pling, the AF phase for strong coupling t ′ /t < .
91, andseveral SL states in-between. There are crucial differ-ences between these new findings and the previous ana-lytical and numerical results . (i) The presence ofthree types of symmetry distinct gapless U (1), SU (2),and Z SL phases that are experimentally distinguish-able. While the Z SL encloses the exact solvable Ki-taev spin model at t ′ /t = 1 and t/U ≪ U (1) SL that separates the Z SL from the SMfor t ′ /t > .
91. The spinon dispersion in the U (1) SLhas the same form as that of the quasiparticle in the SMphase but with renormalized hoppings. For t ′ /t < . U (1) SL transforms with increasing U into the SU (2)SL where a free Dirac fermion spinon dispersion ariseswith t ′ renormalized to zero and the TRS restored. (ii)Our RPA results captures qualitatively several differentAF ordered phases in different parameter regions. Thepresence of the link-spin dependent hopping t ′ introducesa competition between conventional N´eel order and a newtype of AF order accompanied by a local spin rotation .The latter pushes the magnetic phase boundary towardlarger-U dramatically when t ′ /t > .
57, realizing the var-ious symmetry distinct SLs as stable phases of the elec-tronic matter.
U/t t ’ /t Z SL Semi−Metal i-N´eelN´eel
U(1) SL
SU(2) SL
FIG. 5: (color online) Phase diagram. The magnetism partcomes from Fig. 2(b) and the SL part from Fig. 4. We usethe red dash-dot lines to separate the SL and AF phases aswell as i-Neel and Neel because the RPA calculation in thestrong coupling is not as good as that in the weak coupling.
VI. EXPERIMENTAL IMPLICATIONS
To sum up, we studied a Kitaev-Hubbard model usingRPA and slave rotor theory. We obtained a fruitful phasediagram, including semi-metal phase, commensurate andincommensurate AFM ordered phases and three symme-try distinct SL states. We now discuss how to measurethese phases in cold atom experiments.If the SLs proposed are stable, they may be recognizedin cold atom experiments. For example, Bragg spec-troscopy can be used to measure the full band structure(see Fig.3) in cold atoms system . There are some quali-tative difference of the spinon dispersions in the Z , U (1)and SU (2) SL phases. The Z SL differs apparently fromthe other two because there are Majorana fermion exci-tations and non-abelian anyonic Majorana bound statesin an external magnetic field . The U (1) spinon is ofa linear dispersion proportional to t ′ | a zF | at the Diracpoints and does not have a conserved S z , the dispersionof the SU (2) spinon is the same as that of the free Diracfermion with a conserved S z and a renormalized hopping ta F .Bragg spectroscopy can also be used to deter-mine the dynamical spin structure S + − ( ω, q ), whichis the Fourier transformation of spin-spin correlation h S + ( r , t ) S − ( r ′ , t ′ ) i and proportional to the cross sectionof Bragg scattering . In the U (1) SL phase, because ofthe spin flip terms in the effective spinon Hamiltonian, S + − ( ω, q = 0) = 0. In the small ω limit the cross sectionis proportional to the density of states near the Fermisurface (Dirac points), so S + − ( ω, q = 0) ∝ ω for small ω (see Fig.6 Upper panel). In the Z spin liquid phase, S + − ( ω, q = 0) = 0 if ω is smaller than the gap of the(nearly) flat band and a sharp peak appears when the en-ergy transfer is twice the gap (see Fig.6 Lower panel). Incontrast, S + − ( ω, q = 0) = 0 in SU (2) SL phase. Theseproperties can be used to distinguish the SLs.The anti-ferromagnet order can also be measured via Bragg scattering . ω S +− ( ω , q = ) ω S +− ( ω , q = ) FIG. 6: (color online) Upper: Dynamical spin structure factorin U (1) spin liquid phase. U = 3 . t and t ′ = 0 . t . Lower:Dynamical spin structure factor in Z spin liquid phase. U =4 t and t ′ = t AcknowledgementThe authors thank Sen Zhou for useful discussions.This work is supported by the 973 program of MOSTof China (2012CB821402), NNSF of China (11174298,11121403), DOE grant DE-FG02-99ER45747 and NSFDMR-0704545. ZW thanks Aspen Center for Physics forhospitality.
Appendix A: Symmetries of t ′ -term If Ψ A is redefined asΨ A = " c A ↑ − c † A ↓ c A ↓ c † A ↑ then the t ′ -term can be written as − t ′ X h A,B i a Tr(Ψ † A σ a Ψ B )which is pseudo-spin rotational invariant.To reveal the spin-rotational symmetry of t ′ -term, onecan enlarge the unit cell and perform local spin rota-tions of Ψs (see Fig.7): for circle , Ψ → Ψ, for square,Ψ → σ z Ψ, for diamond, Ψ → σ y Ψ, for triangle, Ψ → σ x Ψ. After this rotation, t ′ -term can can be written inspin-rotational invariant way. However, even after thisrotation, t ′ -term can’t be written as the same form as t -term. Because for t ′ -term, electron acquire π phase whenhopping around a hexagon and this phase can not be re-moved by spin rotations.Note that the symmetry operations of t -term and t ′ -term are not compatible, that’s to say, t -term is invariantunder some operations while t ′ -term is invariant underothers, so both symmetries are broken when t and t ′ arenonzero. FIG. 7: (color online) t ′ -term can be written in a spin rota-tional invariant manner if the unit cell is enlarged. Appendix B: Calculation of spin susceptibility
The partition function is Z = R Df † Df e − R β Ldτ ,where L = X f † iσ ( ∂ τ δ ij δ σσ ′ − t σσ ′ ij ) f jσ ′ + U X n i ↑ n i ↓ After performing a Hubbard-Stratonovich transforma-tion in spin channel, we get: Z = R Df † Df Dφe − S and S = Z β dτ X i f † i ∂ τ f i − t X h ij i a f † i ( I + σ a ) f j + U X i φ i + U X i φ i f † i σ z f i Integrating out fermions we get the effective action S eff = U Z β dτ X φ i ( τ ) − Trln[ ∂ τ − t σσ ′ i,j + U σφ i ]Setting φ A = − φ B = φ , then up to second order and in static limit: S eff = P q U [1 − U χ ( q )] φ ( q ) φ ( − q ) where χ ( q ) = − βN P k ,ω n Tr G ( iω n , k ) σ z ⊗ τ z G ( iω n , k + q ) σ z ⊗ τ z and G ( iω n , k ) = iω n −H ( k ) is bare Green’sfunction. If H is diagonalized by a matrix V , i.e., V † ( k ) H ( k ) V ( k ) = diag ( E h k , − E h k , E l k , − E l k ), then χ ( q ) = 12 N X k W k , q W k , − q E h k − q / + E h k + q / + W k , q W k , − q E h k − q / + E l k + q / + W k , q W k , − q E h k + q / + E h k − q / + W k , q W k , − q E l k + q / q + E h k − q / + 12 N X k W k , q W k , − q E l k − q / + E h k + q / + W k , q W k , − q E l k − q / + E l k + q / + W k , q W k , − q E l k − q / + E h k + q / + W k , q W k , − q E l k + q / + E l k − q / (B1)where W ( k , q ) = V † ( k − q / ) σ z ⊗ τ z V ( k + q / ). W ( k , q ) = W † ( k , − q ). Because of the inversion sym-metry, we have V ( − k ) = σ x ⊗ τ V ( k ), then W ( k , q ) = − W ( − k , − q ). Using this relation, Eq.(B1) can be sim-plified: χ ( q ) = 1 N X k (cid:20) W k , q W k , − q E h k − q / + E h k + q / + 2 W k , q W k , − q E h k − q / + E l k + q / + W k , − q W k , q E l k − q / + E l k + q / (cid:21) Note that both anti-ferromagnetic order and ferromag-netic order preserve translational symmetry on honey-comb lattice. Peak at Γ point indicts anti-ferromagneticorder because we have set φ A = − φ B = φ . Appendix C: Mean field theory of SU (2) slave rotortheory1. Z ans¨atz The ans¨atz η † AB = a F σ , η ′† AB,a = a ′ F σ a , η AB = a Z σ , η ′ AB,a = a ′ Z σ a breaks the SU (2) gauge symmetryto Z . As this ans¨atz preserves time reversal symmetry,it works for large U . In this case, the effective Lagrangianreads: L = X i ∈ A,B ( f † iσ ∂ τ f iσ ) + 32 U α X i ∈ A,B ( ∂ τ z ∗ α ∂ τ z α ) − λ X i ∈ A,B ( z ∗ α z α −
1) + 6 tN a F a Z + 6 t ′ N a ′ F a ′ Z − t X h A,B i a F f † Aσ f Bσ − t ′ X h A,B i x a ′ F σf † Aσ f † Bσ − t ′ X h A,B i y a ′ F f † Aσ f † Bσ − t ′ X h A,B i z a ′ F σf † Aσ f Bσ − t X h A,B i a Z z † Aα z Bα + t ′ X h A,B i x a ′ Z z † Aα z † Bβ − t ′ X h A,B i y ia ′ Z z † Aα z † Bβ − t ′ X h A,B i z a ′ Z αz † Aα z Bα Let t ↑ = − ta F f − t ′ a ′ F , t ↓ = − ta F f + t ′ a ′ F ,∆ ↑ = − t ′ a ′ F ( e ik + e ik ) and ∆ ↓ = − t ′ a ′ F ( e ik − e ik ),then eigenvalues of spinons are ε F = ± | t σ ± ∆ σ | ,of rotors are ( ε Z − λ ) = t a Z f f ∗ + 3 t ′ a ′ Z ± q ( t a Z f f ∗ + 3 t ′ a ′ Z ) − | t ′ a ′ Z ( g + g − + 1) − t a Z f | .If a ′ Z t ′ < . ta Z , the minimal of rotor eigenvalues isat Γ point. In the following we assume a ′ Z t ′ < . ta Z and we find this condition is satisfied. The rotorcondensed part is: − ta Z N ( z ∗ A z B + z ∗ A z B ) + t ′ a ′ Z (1 + i )( z A z B + z A z B ) − t ′ a ′ Z ( z ∗ A z B − z ∗ A z B ) + λN z ∗ A/Bα z A/Bα / h.c. and λ min = 3 ta Z + √ t ′ a ′ Z .Let P i ∈ A,B z ∗ iα z iα = 2 z , then z A = z B =2 z/ p √ z B = z A =( i − √ z/ p √
3, so the rotor con-densed part becomes: − ta Z N z − √ t ′ a ′ Z N z +2 λN z .The free energy is( U ′ = 2 αU/ F = − T X k ,i ln (1 + e − βε F,i )+ T X ω n X k ,i ln ( 3 ω n U + ε Z,i ) + const= − T X k ,i ln (1 + e − βε F,i ) + X k ,i p U ′ ε Z,i +2 T X k ,i ln (1 − e − β √ U ′ ε Z,i ) + const (C1)where the constant term is − N λ + 6 tN a F a Z +6 t ′ N a ′ F a ′ Z − ta Z N z − √ t ′ a ′ Z N z + 2 λN z and the second term in the last line is zero point energies of rel-ativistic rotors. Taking derivatives with respect to theparameters we get the following self-consistent equations: ∂F∂a F = X k ,i n f ( ε F,i ) ∂ε F,i ∂a F + 6 tN a Z = 0 ∂F∂a zF = X k ,i n f ( ε F,i ) ∂ε F,i ∂a ′ F + 6 t ′ N a ′ Z = 0 ∂F∂a Z = X k ,i √ U ′ √ ε Z,i coth ( β p U ′ ε Z,i ∂ε Z,i ∂a Z +6 tN ( a F − z ) = 0 (C2) ∂F∂a ′ Z = X k ,i √ U ′ √ ε Z,i coth ( β p U ′ ε Z,i ∂ε Z,i ∂a ′ Z +6 t ′ N ( a ′ F − z / √
3) = 0 ∂F∂λ = X k ,i √ U ′ √ ε Z,i coth ( β p U ′ ε Z,i − N + 2 N z = 0Solving them numerically, we also find two criticallines. When the interaction is smaller than U c we predicta p − wave super-conducting phase which is not reliable.When U > U c we get spin liquid phase, in both casesthe spinons are gapless.In the spin liquid regime, when t ′ /t . .
91 and
U >U c , we get an SU (2) spin liquid with a ′ F = a ′ Z = 0.Otherwise it’s a Z spin liquid (See, e.g., Fig.8(a)).We study the large U limit of Z ans¨atz at t ′ = t = 1 carefully. From Eq.(C2), we know that in thelarge U limit the effective chemical potential of rotorsis λ = U ′ . Writing ε F σ ± = | t σ ± ∆ σ | , and ε ± = (cid:2) t a Z f f ∗ +3 t ′ a ′ Z ± (cid:0) ( t a Z f f ∗ +3 t ′ a ′ Z ) −| t ′ a ′ Z ( g + g − +1) − t a Z f | (cid:1) / (cid:3) / , then in the large U limit Eq.(C2)becomes: a Z = 16 tN X k ∂ε F,σ ± ∂a F , a ′ Z = 16 t ′ N X k ∂ε F,σ ± ∂a ′ F (C3) a F = 124 U ′ tN X k ∂ε Z, ± ∂a Z , a ′ F = 124 U ′ t ′ N X k ∂ε Z, ± ∂a ′ Z , If t ′ = t and U is large enough, a ′ F = a F (See, Fig.8(a)).Then the spinon dispersion becomes: ε F ↑ + = ta F | f | and ε F ↑− = ε F ↓± = ta F . We find two gapless dispersingbands and six gapped flat bands, which matches theexact solution of Kitaev model. In order to be consis-tent with this velocity, the mean field parameter a F inthe mean field spinon dispersion needs to be rescaled to a F = J t . Solving Eq.C3 we find a F U ′ = 0 . t ,which gives α = 0 . t ′ = t , a F = a ′ F , the gappedflat bands acquire a weak dispersion . U/t a F a’ F U/t a oF a zF FIG. 8: (color online) Upper: a F and a ′ F for Z ans¨atz. at t = t ′ = 1. a ′ F /a F → U increases. Lower: a F and a zF for U (1) ans¨atz at t = t ′ = 1. There are two phase transitionsat U c ≈ . t and U c ≈ . t . U (1) ans¨atz The ans¨atz η † AB = a F σ , η ′† AB,a = a zF σ z , η AB = a Z σ , η ′ AB,a = a zZ σ z breaks the SU (2) gauge symmetryto U (1). The spinon Hamiltonian has the same form asthe noninteracting electron Hamiltonian with normalizedhopping in this ans¨atz. We expect it is applicable to asmall U . The symmetry of this ans¨atz is the same as theoriginal model, e.g., time reversal symmetry is broken.As time reversal symmetry restores in the large U limit,this ans¨atz does not work. The effective Lagrangian is: L = X i ∈ A,B f † iσ ∂ τ f iσ + 32 U α X i ∈ A,B ( ∂ τ z ∗ iα ∂ τ z iα )+ λ X i ∈ A,B ( z ∗ iα z iα −
1) + 6 tN a F a Z + 6 t ′ N a zF a zZ − ta F X h A,B i f † Aσ f Bσ − t ′ a zF X h A,B i a f † Aσ σ aσσ ′ f Bσ ′ − ta Z X h A,B i z † Aα z Bα − t ′ a zZ X h A,B i z † Aα σ zαβ z Bβ Dispersion of spinons are ε F = t a F f f ∗ + 3 t ′ a z F ± (cid:2) ( t a F f f ∗ + 3 t ′ a z F ) − | t ′ a z F ( g + g − + 1) − t a F f | (cid:3) / .At Dirac points, the linear dispersion is proportional to t ′ and is not degenerate if SU (2) symmetry is notrestored. When, a zF = 0, the SU (2) gauge symme-try is restored, the dispersion becomes two-fold degen-erate ε = ± t | a F || f | , which is the same as the freeDirac fermion on the honeycomb lattice with the renor-malized hopping t | a F | . The dispersion of rotors are ε Z = λ ±| ( ta Z ± t ′ a zZ ) f | . Rotors may condense at Γ point,and we can write the condensed part explicitly( z A = z B ): − N ta Z ( z + z ) − N t ′ a zZ ( z − z ) + 2 N λ ( z + z ).The free energy is the same form as Eq.C1 with the con-stant term replaced by − N λ + 6 tN a F a Z + 6 t ′ N a zF a zZ − N ta Z ( z + z ) − N t ′ a zZ ( z − z ) + 2 N λ ( z + z ). Since a Z , a zZ >
0, we have z = 0 and λ min = 3( ta Z + t ′ a zZ ).Taking derivatives with respect to the parameters we getthe following self-consistent equations: ∂F∂a F = X k ,i n f ( ε F,i ) ∂ε F,i ∂a F + 6 tN a Z = 0 ∂F∂a zF = X k ,i n f ( ε F,i ) ∂ε F,i ∂a zF + 6 t ′ N a zZ = 0 ∂F∂a Z = X k ,i √ U ′ √ ε Z,i coth ( β p U ′ ε Z,i ∂ε Z,i ∂a Z +6 tN ( a F − z ) = 0 (C4) ∂F∂a zZ = X k ,i √ U ′ √ ε Z,i coth ( β p U ′ ε Z,i ∂ε Z,i ∂a zZ +6 t ′ N ( a zF − z ) = 0 ∂F∂λ = X k ,i √ U ′ √ ε Z,i coth ( β p U ′ ε Z,i − N + 2 N z = 0Solving these self-consistent equations numerically wefind two critical lines: when the interaction is smallerthan ˜ U c we get a semi-metal phase, otherwise the rotorsare gapped and we get a spin liquid phase. We find that a zF and a zZ are strongly suppressed when increasing U and they vanish if U > ˜ U c . In this case we actuallyget an SU (2) spin liquid, see Fig.8(b) . This can beunderstood in the following way: in the large U limitthere is an emergent time reversal symmetry, and thereare two ways to recover this symmetry, that’s, a zF = a zZ =0 or a F = a Z = 0, because we are considering the t ′ < t case, we get a zF = a zZ = 0. This indicates that the U (1)ans¨atz is not reliable in the large U limit. α dependence of spin liquid phase diagram In the main text we choose α = 0 .
572 and get thespin liquid phase diagram Fig.4. We have explained whywe choose the rescale parameter α as t ′ /t -independent.However, one may wonder what the phase diagramlooks like if α is dependent on t ′ /t . The answer is t ′ /t -dependence of α doesn’t change the topology of the spinliquid phase digram because α is only a rescale of the0 U/t t ’ /t Semi−Metal
SU(2) SL
U(1) SL Z SL U/t t ’ /t Semi−Metal
SU(2) SL
U(1) SL Z SL FIG. 9: (color online) α ( t ′ /t = 1) = 0 .
572 and varies linearlywith t ′ /t . Upper: α ( t ′ /t = 0 . .
5. Lower: α ( t ′ /t =1) = 0 . α ( t ′ /t = 0 . . interaction. To show this we plot the phase diagram fordifferent dependence of α on t ′ /t (Fig.9). P. Fazekas and P. W. Anderson, Philos. Mag. , 423(1974). P. W. Anderson, Science , 1196 (1987). P. A. Lee, Science , 1306 (2008). Y. Shimizu, K. Miyagawa, K. Kanoda, M. Maesato, andG. Saito, Phys. Rev. Lett. , 107001 (2003). J. S. Helton, K. Matan, M. P. Shores, E. A. Nytko, B. M.Bartlett, Y. Yoshida, Y. Takano, A. Suslov, Y. Qiu, J.-H. Chung, D. G. Nocera, and Y. S. Lee, Phys. Rev. Lett. ,107204(2007). Y. Okamoto, M. Nohara, H. Aruga-Katori, and H. Takagi,Phys. Rev. Lett. , 137207(2007). T. Itou, A. Oyamada, S. Maegawa, M. Tamura, and R.Kato, Phys. Rev. B , 104413 (2008). T.-H. Han, J. S. Helton, S. Chu, D. G. Nocera, J. A.Rodriguez-Rivera, C. Broholm and Y. S. Lee, Nature ,406(2012). R. H. Colman, C. Ritter, and A. S. Wills, Chem. Mater. , 6897 (2008). H. C. Jiang, Z. Y. Weng, and D. N. Sheng, Phys. Rev.Lett. , 117203 (2008). S. Yan, D. A. Huse, and S. R. White, Science ,1173(2011). H. C. Jiang, H. Yao and L. Balents, Phys. Rev. B ,024424(2012). S. Depenbrock, I. P. McCulloch, and U. Schollwck, Phys.Rev. Lett. , 067201(2012). H. C. Jiang, Z. H. Wang and L. Balents, Nature Physics ,902(2012). P. Fendley, R. Moessner, and S. L. Sondhi, Phys. Rev. B , 214513 (2002). A. Y. Kitaev, Annals. Phys., , 2(2003). A. Y. Kitaev, Annals. Phys., , 2(2006). M. Levin, and X. G. Wen, Phys. Rev. B , 045110(2005). Hong Yao and Steven A. Kivelson, Phys. Rev. Lett. ,247203(2007). Hong Yao and Dung-Hai Lee, Phys. Rev. Lett. ,087205(2011). Z. Y. Meng, T. C. Lang, S. Wessel, F. Assand, and A.Muramatsu, Nature (London) , 847 (2010). S. Sorella, Y. Otsuka, and S. Yunoki, Sci. Rep. , 992(2012). F. F. Assaad and I. F. Herbut, Phys. Rev. X ,031010(2013). W. Wu, Y. -H. Chen, H.-S. Tao, N.-H. Tong, and W.-M.Liu, Phys. Rev. B , 245102 (2010). A. Liebsch, Phys. Rev. B , 035113 (2011). R.-Q. He and Z.-Y. Lu, Phys. Rev. B , 045105 (2012). S. R. Hassan and D. Senechal, Phys. Rev. Lett. ,096402 (2013). A. Liebsch, arXiv: 1302.6547 (2013). Y. Zhou and T. K. Ng, Phys. Rev. B , 165130 (2013). S. Zhou, Y. P. Wang, and Z. Wang, Phys. Rev. B ,195119 (2014).. G. -Y. Sun and S.-P. Kou, J. Phys: Condens. Matter ,045603 (2011). L. M. Duan, E. Demler, and M. D. Lukin, Phys. Rev. Lett. , 090402(2003). C. Zhang, V. W. Scarola, S. Tewari, S. Das Sarma, Proc.Natl. Acad. Sci. USA , 18415 (2007). P. V. Sriluckshmy, A. Mishra, S. R. Hassan, and R.Shankar, Phys. Rev. B , 045105(2014). S. R. Hassan, Sandeep Goyal, R. Shankar, and DavidSnchal, Phys. Rev. B , 045301(2013). S. R. Hassan, P. V. Sriluckshmy, Sandeep K. Goyal,R. Shankar, and David S´en´echal, Phys. Rev. Lett. ,037201(2013). J. P. L. Faye, D. S´en´echal and S. R. Hassan, Phys. Rev. B , 115130(2014). For a recent review , see Z. Nussinov and J van den Brink,arXive 1303.5922. Y. Yu, L. Liang, Q. Niu and S. J. Qin, Phys. Rev. B ,041107(RC) (2013). X. G. Wen,
Quantum Field Theory of Many-Body Systems (Oxford University Press, 2004). K.-S. Kim, Phys. Rev. Lett. , 136402 (2006); Phys. Rev.B , 245105(2007). M. Hermele, Phys. Rev. B , 035125(2007). I. Affleck, Z. Zou, T. Hsu, and P. W. Anderson, Phys. Rev.B , 745(1988). See, for example, N. Nagaosa,
Quantum Field Theory inStrongly Correlated Electronic Systems , (Springer, 1999). The different magnetic properties versus different hoppingprocesses Similar phenomenon was also found in a cor-rected topological insulator model. See, M. Kargarian, A.Langari, and G. A. Fiete, Phys. Rev. B , 205124 (2012). S. Florens and A. Georges, Phys. Rev. B ,035114(2004). S.-S. Lee and P. A. Lee, Phys. Rev. Lett. , 036403(2005). P. A. Lee, N. Nagaosa and X.-G. Wen, Rev. Mod. Phys. , 17(2006). F. J. Burnell and C. Nayak, Phys. Rev. B ,125125(2011). R. Schaffer, S. Bhattacharjee and Y. B. Kim, Phys. Rev.B , 224417(2012). M. Hermele, T. Senthil, M. P. A. Fisher, P. A. Lee, N.Nagaosa, and X.-G. Wen, Phys. Rev. B , 214437 (2004). S.-S. Lee, Phys. Rev. B , 085129 (2008). J. Heinze, S. G¨otze, J. S. Krauser, B. Hundt, N. Fl¨aschner,D.-S. L¨uhmann, C. Becker, and K. Sengstock, Phys. Rev.Lett , 135303 (2011). A. Brunello, F. Dalfovo, L. Pitaevskii, S. Stringari, and F.Zambelli, Phys. Rev. A , 063614 (2001). T. A. Corcovilos, S. K. Baur, J. M. Hitchcock, E. J.Mueller, and R. G. Hulet, Phys. Rev. A , 013415 (2010).56