Excitonic and Nematic Instabilities on the Surface of Topological Kondo Insulators
Bitan Roy, Johannes Hofmann, Valentin Stanev, Jay D. Sau, Victor Galitski
EExcitonic and Nematic Instabilities on the Surface of Topological Kondo Insulators
Bitan Roy, ∗ Johannes Hofmann, Valentin Stanev, and Jay D. Sau
Condensed Matter Theory Center and Joint Quantum Institute,University of Maryland, College Park, Maryland 20742-4111, USA
Victor Galitski
Condensed Matter Theory Center and Joint Quantum Institute,University of Maryland, College Park, Maryland 20742-4111, USA andSchool of Physics, Monash University, Melbourne, Victoria 3800, Australia (Dated: August 17, 2018)We study the effects of strong electron-electron interactions on the surface of cubic topologi-cal Kondo insulators (such as samarium hexaboride, SmB ). Cubic topological Kondo insulatorsgenerally support three copies of massless Dirac nodes on the surface, but only two of them are ener-getically degenerate and exhibit an energy offset relative to the third one. With a tunable chemicalpotential, when the surface states host electron and hole pockets of comparable size, strong interac-tions may drive this system into rotational symmetry breaking nematic and translational symmetricbreaking excitonic spin- or charge-density-wave phases, depending on the relative chirality of theDirac cones. Taking a realistic surface band structure into account we analyze the associatedGinzburg-Landau theory and compute the mean field phase diagram for interacting surface states.Beyond mean field theory, this system can be described by a two-component isotropic Ashkin-Tellermodel at finite temperature, and we outline the phase diagram of this model. Our theory provides apossible explanation of recent measurements which detect a two-fold symmetric magnetoresistanceand an upturn in surface resistivity with tunable gate voltage in SmB . Our discussion can alsobe germane to other cubic topological insulators, such as ytterbium hexaboride (YbB ), plutoniumhexaboride (PuB ). PACS numbers: 73.20.-r, 71.35.Lk
I. INTRODUCTION
It was realized in the past decade that the band struc-ture of a strongly spin-orbit coupled three-dimensionalsolid with preserved time-reversal and inversion symme-tries can be associated with a topological Z index [1, 2].A system with such nontrivial topological index, alsoknown as strong Z topological insulator, belongs to classAII in ten fold way of classification [3]. These mate-rials ideally have an insulating bulk but host an oddnumber of metallic surface states which are protectedagainst time-reversal invariant perturbations. Typicaltopological insulators (such as Bi Se ) are often onlyvery weakly correlated. Our theoretical understandingof these materials is thus based on a noninteracting elec-tronic band structure picture that is not affected by thepresence of weak electron-electron interactions. Withinthe same class (AII), a strongly correlated topologicalKondo insulator (TKI) was predicted to exist in Ref. [4],in which the hybridization between localized f - and con-duction d -electrons opens up a topologically nontrivialbulk-insulating gap below the Kondo temperature. In-deed, a number of recent experiments are strongly sug-gesting that samarium hexaboride (SmB ) possibly sup-ports a TKI below the Kondo temperature ( 50 K) [5–14]. The bulk topological invariant can be computed ∗ Corresponding author: [email protected] within the mean-field description of this system, yieldinga nonzero Z index. These recent findings motivate thesearch for effects where both interactions and topologicaldetails play crucial role at low temperatures [15–17].Motivated by the possibility that TKIs can be a fertileground to support novel interplay of topology and cor-relations, we here consider the effect of strong electronicinteractions on the surface of TKIs and demonstrate thatgapless surface states in these systems can be susceptibletowards nematic and excitonic density-wave phases. Wealso show that our theoretical analysis can be germane totwo recent experiments [18, 19], which could be indica-tive of interaction-induced instabilities on the surface ofa TKI: first, a magnetoresistance measurements on SmB reports a C and C -symmetric magnetoresistance at lowand high temperatures, respectively [18]. These findingsindicate a rotational symmetry breaking nematic order-ing on the surface of a TKI. Second, Ref. [19] reportsa measurement of the surface resistivity in SmB wherethe resistivity increases with varying gate voltage, whichmay, for example, arise due to an underlying excitonicordering. In this work we develop a theory for the in-teracting surface states in TKIs, which provides possibleexplanations to these observations.Consider the typical surface band structure of a cubic topological insulator (for example, SmB ): these systemsare strong Z topological insulators and thus support anodd number of metallic surface states. In the cubic envi-ronment of SmB , the band inversion takes place at thethree X points of the bulk Brillouin zone (BZ) [20, 21]. a r X i v : . [ c ond - m a t . m e s - h a ll ] D ec Hence, an interface of a cubic TI with the vacuum sup-ports three copies of massless Dirac cones at the Γ, X ,and Y points of the surface BZ, as illustrated in Figs. 1(a)and (b) [throughout the paper, we assume that the sur-face is cleaved along a high symmetry axis, such as (001)].The underlying cubic symmetry enforces equal energies E X and E Y of the Dirac nodes at the X and Y points,respectively, which manifests a four-fold rotational C symmetry on the surface. The Γ Dirac point is, however,not constrained by this symmetry and generically dis-plays an offset with respect to the X and Y points, i.e., E Γ (cid:54) = E X/Y (we set E Γ > E X/Y in the remainder to bedefinite), which can be as large as ∼ −
12 meV [22, 23].This surface band structure is also in agreement with re-cent ARPES measurements [8–13, 24–26]. Due to suchlarge energy off-set among the Dirac points, it is naturalto anticipate that surface chemical potential is tuned inbetween E Γ and E X,Y , giving rise to electron and holepockets that can be conducive for excitonic condensation.If, on the other hand, all the pockets are electron or holelike such configuration can be achieved through externalgating, for example [19].It is therefore conceivable to place the chemical po-tential in between E Γ and E X/Y [19], yielding one holepocket around the Γ point and two electron pockets nearthe X , Y points, as shown in Fig. 1(c). Now, if interac-tions on the surface are included, electrons in the X/Y pockets can pair via the so called the Keldysh-Kopaevmechanism with holes in the Γ pockets [28], giving riseto an excitonic condensate i.e., a density wave, which ismodulated by half the reciprocal lattice vector of the sur-face BZ. This paper discusses the phase diagram of thiseffective interacting surface theory.Since, the underlying bulk theory is strongly spin-orbitcoupled, spin (planar components) and momentum of thesurface Dirac cones will be locked as shown in Fig. 1(a)and (b). Therefore, only the z -component of the spinremains free and participates in the ordering. However,in principle, two distinct possible types of excitonic in-stabilities can occur on the surface of TKIs dependingon the relative chirality of the Dirac cones at the X/Y and Γ points. When all Dirac cones on the surface haveidentical chirality [Fig. 1(a)], the excitonic condensate isformed by electrons and holes with opposite spin pro-jection, giving rise to triplet spin-density wave (SDW)order. If, on the other hand, the Dirac points at the Γand
X, Y points carry opposite chirality [Fig. 1(b)], pair-ing occurs between particles and holes with equal spinprojection, leading to singlet charge-density wave (CDW)order. Here singlet and triplet orders are defined in termsof total angular momentum. Our discussion is, however,insensitive to the exact nature of the excitonic ordering,and we thus assume equal chirality for all Dirac conesand discuss the SDW instability in the following.Currently there is an ongoing debate on the effectivemodel for bulk insulating state in SmB that can lead todifferent spin texture on the surface [23, 29]. However,the nature of the excitonic order only depends on the FIG. 1. (Color online) Top row: Two possible chiralities ofelectron (blue) and hole (red) pockets on the surface of cubicTKIs, leading to an excitonic instability in the (a) SDW and(b) CDW channels, respectively. (c) Offset among the Diracpoints near the Γ and
X/Y points, and (d) deviation fromperfectly nesting, due to (e) unequal sizes of the pockets, (f)ellipticity in the electron pocket, parametrized by µ and δ ,respectively. relative chirality of electron- and hole-like Dirac surfaces.Recent theoretical works [30, 31] have demonstrated thatdepending on the relative strength of nearest-neighborand next-nearest-neighbor hybridization among d - and f -electrons, one can realize either two scenarios, we pre-sented in Fig. 1. Therefore, our classification exhaustsall possibilities for the excitonic order and the followingdiscussion is insensitive to the details of the bulk bandstructures (since, the SDW and the CDW orders giveidentical phase diagram).If the Fermi surfaces are perfectly nested [as shown inFig. 1 (d)], the Keldysh-Kopaev mechanism dictates thatan excitonic instability sets in for arbitrarily weak repul-sive interactions. It turns out, however, that a realisticsurface band structure deviates from perfect nesting intwo ways: first, generically the chemical potential will notbe exactly placed in the middle between E Γ and E X/Y [as illustrated in Fig. 1 (e)]. This Fermi surface mismatchreduces the propensity for excitonic pairing, analogous tothe Clogstron-Chandrasekhar effect in standard BCS the-ory where the chemical potential imbalance is induced bya Zeeman term. Second, recent band structure calcula-tions [22] indicate that only the Γ Dirac cone is isotropicwhile both X and Y Dirac cones can be anisotropic [seeFig. 1 (f)], in agreement with ARPES measurements [8–13, 24–26]. We take these realistic effects into account,finding that the overall structure of the phase diagram isnot strongly affected by these effects, although they mayreduce the transition temperature of various orderings.We note that the surface band-structure shown onFig. 1 exhibits strong resemblance to the band structureof the iron-based superconductors [27]. We discuss boththe similarities and the differences between these sys-tems at the end of the paper (see Sec. VI). The excitonicordering due to weak repulsive interactions, known asKeldysh-Kopaev mechanism [28], has also been exploitedto address the SDW instability in Cr [32] and iron-basedsuperconductors [33–35], antiferromagnetic ordering forweak Hubbard repulsion in monolayer [36, 37] and bilayer[38] graphene, 2D Kondo insulators [39] when placed inan in-plane magnetic field, and in the context of possi-ble excitonic instability in topologically trivial calciumhexaboride (CaB ) [40].This paper is structured as follows: in Sec. II, we intro-duce the microscopic description of the interacting sur-face states of a cubic TKI. In Sec. III, we discuss theGinzburg-Landau theory of the model that is valid inthe vicinity of a second order phase transition at finitetemperature. In particular, we discover that in the limitof small ellipticity the order developing on the surfacebreaks the C lattice symmetry down to C . We furtherillustrate that the condensation of excitonic order param-eters only breaks discrete symmetries, thus implying thattrue long range order is described accurately by a meanfield analysis of our effective surface theory. We presenta full numerical computation of the mean field phase di-agram in Sec. IV, finding a second order phase transitionfor nearly perfect nesting from a high-temperature para-magnetic phase to a C -symmetric state at low tempera-ture in which an excitonic condensate develops betweenthe Γ and the either X or Y pockets, but not both, thusspontaneously breaking the discrete C symmetry of thesurface BZ. As is well known, mean field theory does notassume correlations in the paramagnetic or normal phaseat high temperature and does not distinguish between aphase where true long range order develops in the formof a nematic phase (with broken rotational symmetry)where thermal fluctuations dominate [41] and a densitywave phase (with broken translational symmetry). Thesetwo distinct transitions, which coincide in mean field the-ory, can, in principle, take place at different tempera-tures. This occurs through the proliferation of domainwalls in the system. It turns out that the effective the-ory describing the dynamics of the domain walls can bemapped onto a two-component isotropic Ashkin-Teller model. We exploit such mapping in Sec. V to elucidatethe phase diagram beyond the mean field level. Finally,the paper is concluded by a summary and discussion inSec. VI. In particular, we comment on similarities as wellas some differences between our findings and the phasestructure in a completely different class of systems, theiron pnictides.
II. MICROSCOPIC HAMILTONIAN FORINTERACTING SURFACE STATES
This section introduces the microscopic description ofthe interacting surface states. The appropriate spinorbasis is chosen to be Ψ j = (Ψ ↑ ,j , Ψ ↓ ,j ), near j = Γ , X, Y points of the surface BZ, where Ψ σ,j is composed of linearsuperposition of d and f electrons with spin projection σ = ↑ , ↓ . The relative weight among d and f electronsin the surface states is set by the bulk band parame-ters, such as hopping amplitudes and hybridization ma-trix elements [22]. A recent transport measurements inSmB with different thickness clearly establish that a lowtemperatures (sufficiently below the Kondo temperature)surface states are decoupled from the bulk and the trans-port properties are essentially determined by the formerones [6]. Furthermore, spin-resolved ARPES has estab-lished the helical spin-texture of the surface states, andquantum oscillation has observed the signature of DiracLandau levels up to 45 Teslas [42]. Also, recent thermo-electric measurements captured the signature of heavyDirac fermions on the surface, even after mechanicallydamaging the surface [43]. These observations stronglyindicate that despite small bulk gap ( ∼
15 meV), andlarge number of bulk states, the surface and the bulkstates are effectively decoupled in SmB that in turn al-lows us to treat the gapless surface states separately [44].Notice, in YbB the bulk gap is ∼
100 meV and onecan safely neglect any coupling between bulk and surfacestates.The noninteracting Hamiltonians describing the helicalDirac fermionic excitations near the Γ , X and Y pointstake the form [setting (cid:126) = 1] H j = v jx k x σ x − v jy k y σ y , (1)where j = Γ , X, Y , with v Γ x = v Γ y = v as the Fermivelocity of the isotropic Dirac cone near the Γ point.The underlying C symmetry of the surface BZ implies v Xx = v Yy and v Xy = v Yx . The ellipticity of the Diraccones near the X and Y points is captured by defining v Xx = v (1 + δ ) and v Xy = v (1 − δ ). The parameter δ in SmB ranges from 0 . . anti-vortices near Γ , X, Y points of the surface BZ, cap-turing the signature of nontrivial topological invariant ofthe bulk insulating state on the surface.Excitonic SDW ordering arises from a repulsive inter-action between fermions with opposite spin projections inthe Γ and X, Y pockets. Such a particle-hole pairing in-stability can be taken into account by adding a repulsiveshort-ranged interaction H int = − U (cid:88) j = X,Y (cid:90) d q (2 π ) s † j, q s j, q (2)to the free Hamiltonian ( H j ), where U > s j, q = (cid:90) d k (2 π ) c † Γ , k + q α ( σ ) αβ c j, k β (3)is the spin operator. c † j, k α creates a fermion in the j = Γ , X, Y pocket with momentum k and spin α . Themomentum of the X and Y excitation is measured rela-tive to the nesting vectors Q X = ( π,
0) and Q Y = (0 , π ).In Eq. (3), a summation over the spinor indices α and β is implied. Within the same framework, CDW orderingcan be studied by simply replacing the Pauli matrix σ by σ in Eq. (3) and changing the sign of one matrix in H X/Y or H Γ in Eq. (1), without quantitatively changingthe results. The order parameter for the excitonic SDWcondensation is∆ X/Y = U (cid:104) c † Γ , k α ( σ ) αβ c X/Y, k β (cid:105) , (4)where (cid:104) . . . (cid:105) denotes the thermal expectation value. III. GINZBURG-LANDAU THEORY
In this section, we discuss the Ginzburg-Landau ex-pansion of the ordered state, which describes the secondorder phase transition at small δ . This analysis will allowus to gain a qualitative insight into the phase diagram forinteracting surface states of TKIs and the notion of sym-metry breaking in various ordered phases. The Ginzburg-Landau functional can be constructed by systematicallyexpanding the free energy F in powers of ∆ X and ∆ Y ,yielding F (∆ i ) = K (cid:104) ( | (cid:126) ∇ ∆ X | ) + ( | (cid:126) ∇ ∆ Y | ) (cid:105) + α [ | ∆ X | + | ∆ Y | ]+ β | ∆ X | + | ∆ Y | ) + γ | ∆ X | | ∆ Y | . (5)The last term (proportional to γ ) plays an important rolein determining the pattern of symmetry breaking in theordered phase. For γ = 0, the free energy is degeneratefor fixed | ∆ X | + | ∆ Y | . If γ >
0, surface states de-velop a finite expectation value of either | ∆ X | or | ∆ Y | ,but not both. Such a phase manifestly breaks the C rotational symmetry down to C , and the system simul-taneously develops a nematic order. On the other hand,when γ <
0, the system minimizes the free energy bysimultaneously condensing | ∆ X | and | ∆ Y | at the sametemperature, and the four-fold C rotational symmetryof the system is preserved in the ordered phase.In terms of the microscopic parameters, γ reads [49] γ = Tr [ ˆ G Γ ˆ G X ˆ G Γ ˆ G X + ˆ G Γ ˆ G Y ˆ G Γ ˆ G Y − G Γ ˆ G X ˆ G Γ ˆ G Y ] , (6)where we defineˆ G − = − iω + H Γ − λ − , ˆ G − X/Y = − iω + H X/Y + λ + , (7)with λ ± = λ ± µ [see also Fig. 1(c)] and Tr implies asummation over momentum, Matsubara frequency, andspinor indices. If all bands are perfectly circular ( δ = 0),ˆ G X = ˆ G Y and concomitantly γ = 0, which remains trueeven if the bands are not perfectly nested, i.e., µ (cid:54) = 0. In arealistic situation with elliptic eletron-like Fermi pocketsnear X and Y points (i.e., δ (cid:54) = 0), we have γ (cid:54) = 0. Forsmall ellipticity ( δ (cid:28) F in powers of δ , we obtain γ = δ g ( T, µ ), where g ( T c , µ ) is a positive function close to T c . Thus, the SDWstate breaks the C symmetry on the surface under X ↔ Y . In the limit of large ellipticity, we must treat δ non-perturbatively, which is done in the next section.We point out that the Ginzburg-Landau functionalin Eq. (5) possesses a U (1) valley symmetry of theSDW OPs (∆ X , ∆ Y ) associated with their phases ∆ j = | ∆ j | e iφ j , which implies that for γ > rotationalbut also a continuous U(1) symmetry. It is importantto note that such continuous U (1) symmetry is only anartifact of the low energy approximation for the surfacestates and can be reduced if we allow an additional quar-tic term F SB = ρ | ∆ X | | ∆ Y | [cos(2 ϕ X ) + cos(2 ϕ Y )] (8)in Eq. (5). Such a term can, for example, be generated bypair-scattering processes represented by c † Γ c † Γ c X c X and c † Γ c † Γ c Y c Y , also known as Umklapp processes, which areallowed in the presence of an underlying lattice [50]. Thephysical origin of such terms can be appreciated in thefollowing way: the phase degree of freedom of ∆ j rep-resents a sliding mode of the SDW order in real space.However, in any material the commensurate density wavewill be pinned to the lattice. Hence, we need to take intoaccount such lattice-induced terms to pin density-waveorder, that also reduce the (artificial) valley U(1) sym-metry down to a discrete Z one. Most importantly, thisimplies that no continuous symmetry is broken and theSDW order on the two-dimensional surface of cubic TKIscan exhibit true long-range order [51]. In particular, weexpect that a mean field analysis provides an accuratephase diagram of the effective surface theory, despite thefact that the system is two-dimensional. We discuss themean field phase diagram in the next section. IV. MEAN FIELD PHASE DIAGRAM
To go beyond the Ginzburg-Landau regime of thephase diagram, we now analyze the interacting surfacetheory in the mean field approximation. In this sectionwe neglect the symmetry-breaking terms [Eq. (8)], andthus the excitonic orders enjoy an artificial U(1) symme-try. In terms of the order parameters, defined in Eq. (4),the free energy density reads F = 2 U (cid:0) | ∆ X | + | ∆ Y | (cid:1) − β (cid:88) i =1 (cid:90) d k (2 π ) ln (cid:20) βE i (cid:21) , (9)where β is the inverse temperature and E i are thesix eigenvalues of the effective quadratic single-particleHamiltonian H HS = H Γ − λ − σ − ∆ X σ − ∆ Y σ − ∆ † X σ H X + λ + σ − ∆ † Y σ H Y + λ + σ . (10)In the above equation, we set λ ± = λ ± µ as illustrated inFig. 1(a). As is characteristic for two-dimensional Diracsystems, the free energy density in Eq. (9) diverges lin-early due to large-momentum contributions, which, how-ever, can be absorbed in a renormalization of the effectiveinteraction strength 1 U = 1 U − v Λ , (11)where U > U but not onthe non-universal cutoff scale Λ or the bare coupling U .In Fig. 2 (left), we present the phase diagram as ob-tained by minimizing the free energy in Eq. (9) as a func-tion of chemical potential µ and temperature T for thenesting λ = 2 U − and an ellipticity of the X/Y sur-face pockets of δ = 0 .
2. At small chemical potential, theground state displays a two-fold rotational or C symme-try, where electrons from either X or Y pocket pair withholes from the Γ point, respectively, yielding | ∆ X | (cid:54) = 0and | ∆ Y | = 0 or vice versa. The appearance of C SDWorder naturally introduces a nematicity (characterized by∆ X (cid:54) = ∆ Y ) in the system. As the temperature is in-creased, there is a continuous second-order transition outof the SDW phase to the paramagnetic (PM) phase.This limit corresponds to the Ginzburg-Landau anal-ysis presented in the previous section. If the chemicalpotential (and hence the Fermi surface mismatch) is in-creased, the direct C -normal (PM) transition at lowtemperature is masked by an intermediate phase in whichthe C rotational symmetry is restored and all Fermipockets participate in the excitonic pairing. Both C -C and C -PM transitions are first order in nature. Figure 2(right) shows the complete phase diagram as a functionof µ , δ , and T . Increasing the ellipticity δ pushes thecritical chemical potential for the C -C transition tosmaller values but only mildly affects the subsequent C -PM transition. Hence, while a small ellipticity favors theC phase at small µ , the region of the phase diagramwith C -symmetry increases when the Fermi surfaces arestrongly anisotropic.There is an intuitive picture why at small ellipticity thesystem is C symmetric and only at large Fermi surfacemismatch the C symmetric phase arises [47]: for nearlyperfect nesting (small δ and µ ), the same hole-like statenear the Γ point contribute to the excitonic pairing withelectron-like states from X and Y pockets. Thus, pairingbetween Γ and X reduces the available phase space forpairing between Γ and Y and vice versa, implying thatonly one condensate develops and the system enters the C symmetric phase. As the Fermi surface mismatch in-creases, however, disjoint regions of the Γ Fermi surfacecontribute to the excitonic condensation and a C sym-metric phase becomes preferable, as demonstrated by ourfull calculation of the phase diagram.For δ = 0 (circular Fermi surfaces) the quadratic Hamiltonian in Eq. (10) manifests a U(1) symmetryamong the exitonic OPs ∆ X and ∆ Y , and consequentlythe free energy depends only on the magnitude ∆ = | ∆ X | + | ∆ Y | . Thus, in the limit δ = 0, there isno distinction between the C and the C symmetricphases. At zero temperature, the free energy densitythen takes the particularly simple form F = µ − ∆ / represents the SDW OP at T = 0 and µ = 0,which implies a first-order transition between condensedand normal phase at the standard critical Clogston-Chandrasekhar value µ crit = ∆ / √
2, which can also beseen in Fig. 2 (right).We point out that the structure of the BZ in iron-based superconductors is qualitatively similar to the onefor the surface states of cubic TKIs. Interestingly, thephase diagrams of these two systems bear some quali-tative similarities [27, 47, 48]. In particular, the C -C phase transition that can be tuned by doping has beenobserved experimentally in pnictide materials [48].We note that there are two possible ways to modifythe mean field phase diagram: for a large Fermi surfaceanisotropy, the system may condense into an incommen-surate density-wave phase, where the periodicity of theexcitonic condensate is different from the reciprocal lat-tice vector [32]. Furthermore, for large doping, varioussuperconducting instabilities may set in. The discussionof these phenomena is beyond the scope of this paper.Our present mean field analysis does not account forthermal fluctuations. Quite generally, a full analysis ofthe phase diagram should, in principle, distinguish be-tween the nematic and the excitonic phases. As will bediscussed in the following section, once thermal fluctu-ations are incorporated, the transition temperatures forthese two instabilities can be different. Let us focus onthe regime of small chemical potential, where mean-fieldtheory predicts a C phase for arbitrary δ , as shown inFig 2. In this phase either ∆ X or ∆ Y develops a nonzerobut real expectation value and thus the surface states si-multaneously develop a nematic (due to the breaking of C symmetry) as well as a translational symmetry break-ing commensurate SDW order. These orders can be rep-resented by two different Ising-like variables and thus theground state at T = 0 displays an exact four-fold degen-eracy. However, at finite temperature, thermal fluctua-tions allow the system to fragment into multiple domainsof these degenerate phases. We we will argue that inter-play of these domains at finite temperatures can be cap-tured by a two-component isotropic Ashkin-Teller model,and allude to the finite temperature phase diagram forthe surface states beyond the mean field approximation.Before concluding the section a discussion on the na-ture of the nematic order seems appropriate. Notice thatthe nematic phase is described by a fluctuating excitonicorder that does not acquire a finite vacuum expectationvalue. As pointed out in the Introduction that depend-ing on the relative strength of nearest-neighbor and next-nearest-neighbor hybridization amplitude among the op-posite parity orbitals (such as d and f ) in the bulk, the FIG. 2. (Color online) Left: phase diagram for λ = 2 U − and δ = 0 . µ andtemperature T . At small temperature and chemical potential, the ground state has only a C symmetry. Red (thick) and blue(thin) lines denote second and first order phase transitions, respectively. Right: Phase diagram for λ = 2 U − as a functionof Fermi surface anisotropy δ , µ , and T . Notation as in the left panel. The parameter δ in various ARPES experiments are δ = 0 .
25 [8], 0 .
11 [11], 0 .
21 [12], 0 .
33 [13]. It is worth pointing out that the phase diagrams we obtain here are qualitativelysimilar to the one extracted experimentally for iron pnictides, which also share similar structure of the BZ [48]. chiralities of electron and hole pockets can be same oropposite, which in turn determines the nature of density-wave excitonic order (SDW or CDW). Therefore, depend-ing on bulk hybridization strength over a finite range, thenematic phase may represent either a fluctuating charge-or spin-density-wave order. However, the phase diagramof the interacting surface states is insensitive to the exactnature of the ordering, as only discerte Ising-like sym-metries are broken in the charge- or spin-density-wavephases (uniform or fluctuating).
V. THERMAL FLUCTUATIONS, DOMAINWALLS AND ASHKIN-TELLER MODEL
To understand the role of a domain walls at finite tem-peratures, we first consider a simpler situation, where thesystem exhibits only a two-fold degeneracy among theconfigurations, say A and B [chosen from four possiblestates with ∆ X > X < Y > Y < F = J AB − T S AB , where S AB ( J AB ) isthe entropy (energy) per unit length of a single domainwall. For temperatures T > J AB /S AB , we have F < s = sgn( | ∆ X | − | ∆ Y | ) and σ = sgn(∆ X + ∆ Y ).The spin variable s determines the direction of the SDWorder, while σ represents how the translation symmetryis broken. Therefore, in the nematic phase s (cid:54) = 0, andwhen the density-wave order condenses we have σ (cid:54) = 0. The energy of the domain walls can be accounted for byan effective exchange Hamiltonian H ex = − (cid:88) (cid:104) i,j (cid:105) [ J s i s j + J (1 + s i s j ) σ i σ j ] , (12)where J represents the energy a domain wall betweenthe regions where | ∆ X | (cid:54) = 0 and | ∆ Y | (cid:54) = 0. J representsa similar quantity where ∆ X or ∆ Y changes the signwithout changing the direction of the symmetry breaking(hence the factor (1 + s i s j )). We expect J /J to beproportional to δ , where δ is the ellipticity of the pocketsnear X and Y points. In terms of a redefined variable s → ˜ s = sσ , the rescaled Hamiltonian assumes the formof a two-component isotropic Ashkin-Teller model [52] H ex = − J (cid:88) (cid:104) i,j (cid:105) (˜ s i ˜ s j + σ i σ j ) − J (cid:88) (cid:104) i,j (cid:105) ˜ s i ˜ s j σ i σ j . (13)The phase diagram of this model is shown in Fig. 3 [53],which we discuss below qualitatively in terms of the orig-inal variables s and σ .For weak Fermi surface anisotropy, which correspondsto small values of J /J ( ∼ δ ), there exists a continu-ous transition (across the dashed line in Fig. 3) from ahigh temperature disordered phase to a low temperatureordered phase. Along this line of direct transition be-tween the disordered and the ordered phases, the expo-nents change continuously , much like for the Kosterlitz-Thouless transition. In the ordered phase, the surfacestates break both translational (by the SDW order) androtational (by the nematic order) symmetries, and the ex-pectation values of the Ising-spin variables in Eq. (12) are (cid:104) s (cid:105) (cid:54) = 0 and (cid:104) σ (cid:105) (cid:54) = 0. This phase is also known as the Bax-ter phase [52]. However, for large δ or J /J (large Fermi FIG. 3. Phase diagram of the two-component isotropicAshkin-Teller Model [53]. In terms of microscopic parame-ters J ∼ U and J /J ∼ δ . surface anisotropy) transitions associated with these twosymmetry breakings bifurcate and occur at distinct tem-peratures. The system first condenses into the nematicphase, where (cid:104) s (cid:105) (cid:54) = 0 but (cid:104) sσ (cid:105) = (cid:104) σ (cid:105) = 0, and onlysubsequently enters the ordered (Baxter) phase at lowertemperature. Next we characterize each of these phasesin terms of original order parameters, ∆ X and ∆ Y .The nematic phase is ordered along either Q X = ( π, Q Y = (0 , π ) in such a way that a large density ofsign flips (domain wall) of the order parameter prolif-erate in the system. In this phase (cid:104)| ∆ X |(cid:105) or (cid:104)| ∆ Y |(cid:105) isnon-zero, but (cid:104) ∆ X (cid:105) = (cid:104) ∆ Y (cid:105) = 0. Consequently, thenematic phase breaks the C rotational symmetry, yetstill retains the translational invariance of the the system.Only at lower temperature, through a subsequent transi-tion system enters into the ordered/Baxter phase, whereboth nematic and density-wave orders develop non-zeroexpectation value. It is worth mentioning that a simi-lar, but distinct, nematic phase has also been studied foriron-based superconductors [49, 54, 55]. VI. SUMMARY AND DISCUSSION
In summary, we discuss various many-body instabili-ties on the surface of strongly interacting cubic TKIs. Weshow that if the chemical potential is placed in betweenthe Dirac points at the Γ and
X/Y points of the surfaceBZ and the resulting electron (near X and Y points)and hole (near Γ point) pockets are of comparable size,fermions can condense into a nematic and density-wavephase. In this phase only one of the electron pocketsparticipates in excitonic pairing, and thus the 4-fold ro-tation symmetry on the surface gets lifted spontaneously.Therefore, our results provide a possible explanation forthe recently observed C symmetric magnetoresistence[18] and the upturn in surface resistivity with tunablegate voltage or equivalently the chemical potential [19]in SmB . The excitonic phase, however, can display a spin- orcharge-density wave ordering depending on the relativechirality of the Dirac cones with electron and hole likecarriers. In Sec. II, we argued that due to the pres-ence of underlying strong spin-orbit coupling that causesspin(in-plane components)-momentum locking of the sur-face states [1, 2], and only the z -component of electrons’spin participates in various instabilities, which in turnalso allows the system to exhibit long-range order atfinite- T . Our results are substantiated by complimentaryGinzburg-Landau analysis of order parameters (for smallFermi surface mismatch) in Sec. III and the free-energyminimization in mean-field approximation (for arbitraryvalues of the parameters µ, δ, λ, T ) in Sec. IV. For largeFermi mismatch, on the other hand, our mean-field anal-ysis predicts that both electron pockets gets involved inexcitonic ordering, and the ordered phase restores the4-fold rotational symmetry of the surface BZ.Furthermore, we extend our analysis beyond the mean-field level, and account for thermal fluctuations and do-main walls when the system condenses into a C density-wave phase in Sec. V. In this limit, the system can bedescribed by a two-component isotropic Ashkin-Tellermodel and we presented a finite temperature phase di-agram in Fig. 3. For small Fermi surface mismatch, bothnematic and density-wave orders condense at the sametemperature in agreement with our mean-field analysis.Only for substantial Fermi surface mismatch, these twotransitions take place at different temperatures. Systemfirst pairs into a nematic phase and yet at a lower temper-ature to an excitonic (Baxter) phase. Although our studyis primarily motivated by ongoing experimental worksin SmB [18, 19] that are suggestive of the presence ofstrong electronic correlations on its surface, it can de-scribe various signature of electron-electron interactionson the surface of other strongly interacting cubic TIs,such as YbB [24–26], PuB [56].Various recent experiments have extracted the effec-tive parameters for the surface band structure. For ex-ample, ARPES experiments have found the ellipticityfactor δ = 0 . − . .Extracting the energy offset among the Dirac points inan experiment is a challenging task. Nevertheless, var-ious first-principle [23] and effective band structure [22]calculations suggest that | E Γ − E X/Y | ∼ −
10 meV. Theestimated values of these parameters indicates that whilethe ellipticity of the Fermi pockets is not too large to de-stroy the propensity of nematic and excitonic orderingson the surface, a large offset among the Dirac points allowone to tune the surface chemical potential over a reason-ably wide range to realize electron and hole pockets ofcomparable sizes thourhg external gating [19], conducivefor orderings. Therefore, with currently estimated valuesof these band parameters, it is quite conceivable that sur-face states of SmB or other cubic TKIs (such as YbB and PuB ) can accommodate various exotic broken sym-metry phases.Detection of the nematic or the C symmetric excitonicorderings demands direction dependent measurements oftransport quantities, for example. Here, we focus onlyon the C symmetry breaking ordering, as it occupiesmost of the phase diagram in Fig. 2. Notice in the ne-matic and the excitonic phases the four-fold rotationalsymmetry gets broken, while the former one is deviodof uniform condensation of any order parameter. There-fore, to pin the the onset of these orderings one needsto perform direction dependent measurements of variousphysical quantities, such as conductivity, resistivity, mag-netoresistence, on the surface that can sense the lack ofrotational symmetry in the close proximity to an order-ing. Recent experiment [18] has reported the lack of fourfold rotational symmetry in magnetoresistance in SmB below 5K, which is suggestive of at least a nematic or-dering on the surface.It should be noted that the surface BZ of cubic TKIclosely is similar to the one in pnictides [49, 54, 55]. How-ever, there exist several crucial differences between thesetwo systems. For example, due to the strong spin-orbitcoupling the SDW order of the surface states breaks onlythe discrete Z symmetry [note that the valley U (1) sym-metry of SDW order is only an artifact of the low energyapproximation in Eqs. (1), (10) which gets reduced to Z due to the presence of an underlying lattice captured bythe term F SB in Eq. (8)], responsible for true long-rangeorder, whereas spin-rotation is a good symmetry and theSDW phase breaks continuous SO (3) symmetry in pnic-tides [54]. In addition, the Fermi surfaces on the surfaceof TKIs constitute vortices or anti-vortices in momentumspace that in turn encode the bulk topological invarianceof the system, while the bands in pnictide materials areregular non-relativistic parabolic bands. Consequently,the regular parabolic bands and therefore the SDW or-der in pnictides can carry additional orbital degeneracy,which depends on various nonuniversal details of the sys-tem [49], whereas the non-interacting model [see Eq. (1)]and the SDW/CDW order [see Eq.(10)] we consider forthe surface states TKIs is constrained by nontrivial bulktopological invariant. Thus, neither SDW nor CDW isaccompanied by additional degeneracy. In additional,contrast to our results, a recent theoretical study findsthat the transition from paramagnetic- C SDW in pnic-tide is discontinuous or first order in nature [57]. Despitethese fundamental differences, we find that the qualita- tive structure of the phase diagram in Fig. 2 for the sur-face states of TKIs bears some similarities to the one foriron pnictides both calculated theoretically [49] and alsowith the one obtained experimentally (see the phase di-gram of Ba − x Na x Fe As in Fig. 2 of Ref.[48]). The sim-ilarity between such different systems is both surprisingand encouraging. Therefore, we expect that our studywill initiate future works related to TKIs that may un-earth some exotic effects due to the presence of strongelectronic correlations in these systems and may as wellshed light into the phase diagram of iron pnictides.As a final remark, we highlight some other possiblephenomena, arising from strong residual electronic inter-actions on the surface of TKIs, among which the renor-malization of plasmon spectrum due to strong fluctua-tions [15], non-Fermi liquid phase for d -electrons [16],quasi-particle inteference [29], and spontaneous valleyHall ordering in the presence of strong magnetci field [58].In addition, a spatial variation of the hybridization hasbeen proposed to lead to a topological chiral-liquid on thesurface [59], without destroying the helical structure ofthe surface states (protected by bulk topological invari-ant). While these proposals are quite fascinating and ofdefinite fundamental importance, our work focuses on thepossibilities of realizing various broken symmetry phases(excitonic and nematic) on the surface of TKI, resultingfrom strong residual interactions, which can explain somepeculiar experimental observations in recent past [18, 19]. ACKNOWLEDGMENTS
We thank P.M.R. Brydon for useful discussions. B.R.and J.D.S. were supported by the startup grant of J.D.Sby University of Maryland. J.H. was supported by LPS-CMTC. V.S. and V.G. were supported by DOE-BESDESC0001911, and Simons Foundation. B. R. is thank-ful to the Max-Planck Institute for Complex Systems,Dresden, Germany, for hospitality during the workshop“Topology and Entanglement in Correlated QuantumSystems (2014)”, where part of this work was completed.V. G. is grateful to Johnpierre Paglione and MichaelFuhrer for useful discussions. [1] M. Z. Hassan, and C. L. Kane, Rev. Mod. Phys. , 3045(2010), and references therein.[2] X. L. Qi, and S.-C. Zhang, Rev. Mod. Phys. , 1057(2011), and references therein.[3] A. P. Schnyder, S. Ryu, A. Furusaki, A. W. W. Ludwig,Phys. Rev. B , 195125 (2008).[4] M. Dzero, K. Sun, V. Galitski and P. Coleman, Phys.Rev. Lett. , 106408 (2010); M. Dzero, K. Sun, P.Coleman and V. Galitski, Phys. Rev. B , 045130(2012). [5] S. Wolgast, C. Kurdak, K. Sun, J. W. Allen, D-J. Kim,and Z. Fisk, Phys. Rev. B , 180405 (2013).[6] D. J. Kim, S. Thomas, T. Grant, J. Botimer, J. Fisk, J.Xia, Sci. Rep. , 3150 (2013).[7] X. Zhang, N. P. Butch, P. Syer, S. Ziemak, R. L. Greene,and J. Paglione, Phys. Rev. X , 011011 (2013).[8] M. Neupane, N. Alidoust, S-Y. Xu, T. Kondo, Y. Ishida,D. J. Kim, Chang Liu, I. Belopolski, Y. J. Jo, T-R.Chang, H-T. Jeng, T. Durakiewicz, L. Balicas, H. Lin, A.Bansil, S. Shin, Z. Fisk, and M. Z. Hasan, Nat. Comm. , 2991 (2013).[9] J. Jiang, S. Li, T. Zhang, Z. Sun, F. Chen, Z.R. Ye, M.Xu, Q.Q. Ge, S.Y. Tan, X.H. Niu, M. Xia, B.P. Xie, Y.F.Li, X.H. Chen, H.H. Wen, and D.L. Feng, Nat. Comm. , 3010 (2013).[10] N. Xu, P. K. Biswas, J. H. Dil, R. S. Dhaka, G. Landolt,S. Muff, C. E. Matt, X. Shi, N. C. Plumb, M. Radovic,E. Pomjakushina, K. Conder, A. Amato, S.V. Borisenko,R. Yu, H.-M. Weng, Z. Fang, X. Dai, J. Mesot, H. Ding,M. Shi, Nat. Comm. , 4566 (2014).[11] J. Jiang, S. Li, T. Zhang, Z. Sun, F. Chen, Z. R. Ye, M.Xu, Q. Q. Ge, S. Y. Tan, X. H. Niu, M. Xia, B.P. Xie, Y.F. Li, X. H. Chen, H. H. Wen, D. L. Feng, Nat. Commun. ,3010 (2013).[12] C-H. Min, P. Lutz, S. Fiedler, B. Y. Kang, B. K. Cho,H.-D. Kim, H. Bentmann, F. Reinert, Phys. Rev. Lett. , 226402 (2014).[13] J. D. Denlinger, J. W. Allen, J.-S. Kang, K. Sun, J.-W. Kim, J. H. Shim, B. I. Min, D.-J. Kim, Z. Fisk,arXiv:1312.6637.[14] D. J. Kim, J. Xia, and Z. Fisk, Nat. Materials , 466(2014).[15] D. K. Efimkin, V. Galitski, Phys. Rev. B , 081113(R)(2014).[16] P. Nikoli´c, Phys. Rev. B , 235107 (2014).[17] H. K. Pal, V. I. Yudson, D. L. Maslov, Phys. Rev. B ,085439 (2012).[18] Z. Yue, X. Wang, D. Wang, J. Wang, D. Culcer, S. Dou,J. Phys. Soc. Jpn. , 044717 (2015).[19] P. Syers, D. Kim, M. S. Fuhrer, J. Paglione, Phys. Rev.Lett. , 096601 (2015).[20] V. Alexandrov, M. Dzero, and P. Coleman, Phys. Rev.Lett. , 226403 (2013).[21] T. Takimoto, J. Phys. Soc. Jpn. , 123710 (2011).[22] B. Roy, J. D. Sau, M. Dzero, and V. Galitski, Phys. Rev.B , 155314 (2014).[23] R. Yu, H. Weng, X. Hu, Z. Fang, and X. Dai, New J.Phys. , 023012 (2015).[24] M. Xia, J. Jiang, Z. R. Ye, Y. H. Wang, Y. Zhang, S. D.Chen, X. H. Niu, D. F. Xu, F. Chen, X. H. Chen, B. P.Xie, T. Zhang, D. L. Feng, Sci. Rep. , 5999 (2014).[25] M. Neupane, S-Y. Xu, N. Alidoust, G. Bian, D-J. Kim,C. Liu, I. Belopolski, T-R. Chang, H-T. Jeng, T. Du-rakiewicz, H. Lin, A. Bansil, Z. Fisk, M. Z. Hasan, Phys.Rev. Lett. , 016403 (2015).[26] N. Xu, C. E. Matt, E. Pomjakushina, J. H. Dil, G. Lan-dolt, J.-Z. Ma, X. Shi, R. S. Dhaka, N. C. Plumb, M.Radovic, V. N. Strocov, T. K. Kim, M. Hoesch, K. Con-der, J. Mesot, H. Ding, M. Shi, arXiv:1405.0165.[27] I. Eremin and A. V. Chubukov, Phys. Rev. B , 024511(2010).[28] L. V. Keldysh and Y. V. Kopaev, Sov. Phys. Solid State , 2219 (1965).[29] P. P. Baruselli and M. Vojta, Phys. Rev. B , 201106(R)(2014).[30] M. Legner, A. R¨uegg, M. Sigrist, Phys. Rev. Lett. ,156405 (2015).[31] P. P. Baruselli, M. Vojta, Phys. Rev. Lett. , 156404(2015).[32] T. M. Rice, Phys. Rev. B , 3619 (1970).[33] V. Cvetkovic, Z. Tesanovic, Eur. Phys. Lett. , 37002(2009); Phys. Rev. B , 024512 (2009).[34] A. V. Chubukov, D. V. Efremov, and I. Eremin, Phys.Rev. B , 134512 (2008). [35] Q. Han, Y. Chen, and Z. D. Wang, Europhys. Lett. ,37007 (2008).[36] I. L. Aleiner, D. M. Kharzeev, and A. M. Tsvelik, Phys.Rev. B , 195415 (2007).[37] M. Bercx, T. C. Lang, and F. F. Assaad, Phys. Rev. B , 045412 (2009).[38] B. Roy, and K. Yang, Phys. Rev. B , 241107(R) (2013).[39] I. Milat, F. F. Assaad, M. Sigrist, Eur. Phys. J. B ,571 (2004).[40] M. E. Zhitomirsky, T. M. Rice, V. I. Anisimov, Nature
251 (1999); V. Barzykin, and L. P. Gorkov, Phys.Rev. Lett. , 2207 (2000); L. Balents and C. M. Varma, ibid. , 1264 (2000).[41] For nematic ordering in parabolic semimetals in two di-mensions see K. Sun, H. Yao, E. Fradkin and S. A. Kivel-son, Phys. Rev. Lett. , 046811 (2009); Y. You, andE. Fradkin, Phys. Rev. B , 235124 (2013), and in threedimensions consult H-H. Lai, B. Roy, and P. Goswami,arXiv:1409.8675.[42] G. Li, Z. Xiang, F. Yu, T. Asaba, B. Lawson, P. Cai,C. Tinsman, A. Berkley, S.Wolgast, Y. S. Eo, Dae-JeongKim, C. Kurdak, J. W. Allen, K. Sun, X. H. Chen, Y. Y.Wang, Z. Fisk, and Lu Li, Science , 1208 (2014).[43] Y. Luo, H. Chen, J. Dai, Z-a. Xu, J. D. Thompson, Phys.Rev. B , 075130 (2015).[44] Chemical potential in SmB resides inside the bulkKondo insulating gap. Hence, SmB represents a true topological insulator and below the Kondo temperaturesurface states decouple from the bulk. Only the statesthat are placed far from the Fermi energy can, in princi-ple, couple with surface states. However, states far fromthe Fermi energy are not sharply defined as the quasi-particle lifetime for these states is finite (standard pic-ture of Fermi liquid theory). Therefore, despite a largenumber of bulk states, decoupling of the sharp surfacestate with infinite quasi-particles life time is a valid ap-proximation, due to its true insulating nature of SmB ,sufficiently below the Kondo temperature.[45] J. Hofmann, Phys. Rev. D , 125027 (2010).[46] B. Roy, M. P. Kennett, S. Das Sarma, Phys. Rev. B ,201409(R) (2014).[47] P. M. R. Brydon, J. Schmiedt, C. Timm, Phys. Rev. B , 214510 (2011).[48] S. Avci, O. Chmaissem, J.M. Allred, S. Rosenkranz,I. Eremin, A.V. Chubukov, D.E.Bugaris, D.Y. Chung,M.G. Kanatzidis, J.-P Castellan, J.A. Schlueter, H.Claus, D.D. Khalyavin, P. Manuel, A. Daoud-Aladine,and R. Osborn, Nature Communications , 3845 (2014).[49] For an analogous calculation in the case of iron-basedsuperconductors see R. M. Fernandes, A. V. Chubukov,J. Knolle, I. Eremin, J. Schmalian, Phys. Rev. B ,024534 (2012).[50] For lattice-induced U(1) symmetry breaking terms ingraphene see B. Roy, and I. F. Herbut, Phys. Rev. B , 035429 (2010).[51] N. D. Mermin and H. Wagner, Phys. Rev. Lett. , 1133(1966); ibid. , 1133 (1966).[52] R. J. Baxter, Exactly Solved Models in Statistical Me-chanics (Academic Press Limited, London, 1982).[53] R. V. Ditzian, J. R. Banavar, G. S. Grest, L. P. Kadanoff,Phys. Rev. B , 2542 (1980), and references therein.[54] C. Xu, M. Muller, S. Sachdev, Phys. Rev. B , 020501(R) (2008). [55] For more references and an extensive discussion of thepossible origin and manifestation of the nematic phase iniron-based superconductors see R. M. Fernandes and J.Schmalian, Supercond. Sci. Technol. , 084005 (2012).[56] X. Deng, K. Haule, G. Kotliar, Phys. Rev. Lett. ,176404 (2013). [57] J. Kang, X. Wang, A. V. Chubukov, R.M. Fernandes,Phys. Rev. B , 121104(R) (2015).[58] X. Li, B. Roy, and S. Das Sarma, arXiv:1507.06647[59] V. Alexandrov, P. Coleman, O. Erten, Phys. Rev. Lett.114