Observation of robust zero-energy state and enhanced superconducting gap in a trilayer heterostructure of MnTe/Bi2Te3/Fe(Te, Se).

The interface between magnetic material and superconductors has long been predicted to host unconventional superconductivity, such as spin-triplet pairing and topological nontrivial pairing state, particularly when spin-orbital coupling (SOC) is incorporated. To identify these unconventional pairing states, fabricating homogenous heterostructures that contain such various properties are preferred but often challenging. Here, we synthesized a trilayer-type van der Waals heterostructure of MnTe/Bi2Te3/Fe(Te, Se), which combined s-wave superconductivity, thickness-dependent magnetism, and strong SOC. Via low-temperature scanning tunneling microscopy, we observed robust zero-energy states with notably nontrivial properties and an enhanced superconducting gap size on single unit cell (UC) MnTe surface. In contrast, no zero-energy state was observed on 2-UC MnTe. First-principle calculations further suggest that the 1-UC MnTe has large interfacial Dzyaloshinskii-Moriya interaction and a frustrated AFM state, which could promote noncolinear spin textures. It thus provides a promising platform for exploring topological nontrivial superconductivity.

INTRODUCTION

When materials with different quantum orders meet at the interface, new physical phenomena may emerge and bring potential applications. A well-known example is the ferromagnet (s-wave)–superconductor interface. Intensive studies have shown that while the spin-singlet pairing potential decays fast in ferromagnet, an odd-frequency spin-triplet pairing state can be induced (, ). A spin rotation layer or spin-orbital coupling (SOC) can further stabilize triplet Cooper pair and generate spin-polarized supercurrent (–), which is of particular interest in spintronic applications. Recently, there is increased research on magnet-superconductor interface with strong SOC or noncolinear spin textures, as these systems may give rise to topological superconductivity (TSC) (–). The chiral magnetic atom chain grown on strong SOC superconductors was considered as its one-dimensional (1D) case (, ), which is in analogy to the semiconductor nanowire–superconductor hybrids under Zeeman field (, ). In the 2D case, magnetic moments arranged on superconductor can form Shiba lattice (, ). Various types of chiral spin textures are predicted to induce effective p-wave pairing, and Majorana boundary mode will emerge when the system enters topological phase. The merits of these potential TSC platforms are that they do not require external field, and the spin textures could be controlled electronically (, ). Besides, theoretical works also predicted that if a 3D topological insulator (TI) layer is inserted in between ferromagnet and superconductor layers, 2D TSC/chiral Majorana edge mode can be induced because of different gap opening in the top/bottom surface states (, ).

To experimentally identify these unconventional pairing states, fabrication of high-quality magnet-superconductor heterostructures with proper interface is highly desired since unconventional pairing is often sensitive to disorder or inhomogeneity, and local probe study such as scanning tunneling microscopy (STM) favors atomically flat surface/interface. Several recent STM studies (–) have demonstrated the advantage of using layered magnetic material (such as transition metal chalcogenides) to make these heterostructures, as they can be grown epitaxially and persist magnetism down to single-layer limit. Moreover, it is also preferred to introduce strong SOC material or topological band structure (such as a TI) into the heterostructure, as they can further promote the TSC phase. In particular, it was shown that if combining strong SOC and inversion asymmetry (which always happens at interface) in a magnetic system, then large Dzyaloshinskii-Moriya interactions (DMI) can be induced and give rise to noncolinear spin textures (–). Transport studies on ferromagnet-TI interfaces have observed signatures of skyrmion-like spin structure (–). Therefore, it would be very promising to fabricate a magnet-superconductor heterostructure incorporated with strong SOC materials for exploring potential unconventional superconductivity. However, so far, this kind of heterostructure is rarely reported.

In this work, we successfully synthesized a trilayer-type heterostructure of MnTe/Bi2Te3/Fe(Te, Se) that has atomically sharp interface. It combines the magnetism of MnTe, strong SOC from Bi2Te3 and superconductivity from Fe(Te, Se). Via low-temperature STM, we observed zero-energy conductance peaks (ZECP) on many regions of 1–unit cell (UC)–thick MnTe surface and an enlarged superconducting gap in comparison to that of Fe(Te, Se). The ZECPs show several distinguished properties: They are not correlated to the observed surface defects, robust under out-of-plane field, and display saturated conductance at low tunneling barriers. Meanwhile, no in-gap state was observed on 2-UC MnTe surface. Our first-principle calculations suggest that 1-UC MnTe with the Bi2Te3 host large interfacial DMI and a frustrated antiferromagnetic (AFM) state, which can promote noncolinear spin textures. Thus, the emergence of ZECPs and their nontrivial features may originate from topological pairing state.

RESULTS

Growth and STM characterization of the heterostructure

MnTe is a layered AFM insulator with a NiAs-type hexagonal structure (which has an in-plane ferromagnetic order and AFM coupling between each plane) (, ), while bulk Bi2Te3 is a 3D TI whose UC has a “quintuple layer” structure along c axis (). We fabricated the MnTe/Bi2Te3/Fe(Te, Se) heterostructure via ultrahigh vacuum molecular beam epitaxy (UHV-MBE). Bi2Te3 (1 UC) was firstly grown on as-cleaved Fe(Te, Se) surface, followed by the growth of 1- to 2-UC MnTe film and post-annealing. The sample was then transferred to a low-temperature STM with calibrated resolution (fig. S1). Detailed experimental procedures are described in Materials and Methods. The 1-UC Bi2Te3 here not only provides strong SOC to the interface but also acts as a necessary buffer layer for MnTe growth, as its in-plane lattice constant (0.437 nm) is close to MnTe (0.419 nm) and has a Te-terminated surface. Moreover, previous studies have shown that Bi2Te3 has good superconducting proximity effect when grown on Fe(Te, Se) (, ).

Figure 1A shows a large-scale STM image of as-grown MnTe/Bi2Te3/Fe(Te, Se) heterostructure. The surface contains various types of atomically flat terraces, including 1-UC Bi2Te3, 1- or 2-UC MnTe on 1-UC Bi2Te3, and bare Fe(Te, Se) substrate. They can be identified through the step height (reflected by different colors in Fig. 1A) and the in-plane lattice structure. Figure 1B shows a line profile taken across several terraces in Fig. 1A. The 1-UC Bi2Te3 has its characteristic height of ~1.0 nm, and 1-UC MnTe has a height of 0.32 nm, which corresponds to the single UC thickness in their bulk form (, ). Panels C and D of Fig. 1 are zoomed-in images of 1-UC Bi2Te3 and 1-UC MnTe surfaces, while panels E and F of Fig. 1 show their atomic lattice, respectively. Both of the two surfaces have a hexagonal lattice with a constant of 0.43 nm and are oriented in the same direction, which indicates that MnTe is epitaxially grown on Bi2Te3 (sketched in Fig. 1B). Meanwhile, the exposed Fe(Te, Se) remains intact with a square lattice (Fig. 1G), indicative of a sharp interface. We note that this MnTe/Bi2Te3 epitaxial structure is different from the MnBi2Te4 film reported in (), which is also grown by MBE but with different growth/annealing conditions. A similar epitaxial structure was reported in the growth of MnSe on Bi2Se3 surface (). As seen in Fig. 1 (C and D), there are certain amounts of point defects on both Bi2Te3 and MnTe surfaces, which locate at the hollow site of Te lattice, as indicated in Fig. 1 (E and F). They are thus likely the substitutional defects of Bi or Mn (). Nonetheless, these defects do not affect the superconductivity in either MnTe or Bi2Te3 surface noticeably, as shown in fig. S2 and further discussion below.

Fig. 1.

The surface of MnTe/Bi2Te3/Fe(Te, Se) heterostructure.

(A) Typical topographic image of the heterostructure (160 nm by 160 nm, Vb = 1 V, I = 20 pA). Different surface termination layers can be identified by different false colors. (B) A line profile taken along the dashed line in (A). The lattice structure of MnTe and Bi2Te3 layers and Fe(Te, Se) substrate are illustrated. (C and D) Topographic image of the 1-UC Bi2Te3 (Vb = 1 V, I = 100 pA) and 1-UC MnTe/Bi2Te3 surface (Vb = 1 V, I = 20 pA), respectively. (E to G) Atomically resolved images of MnTe, Bi2Te3, and exposed Fe(Te, Se) surface, respectively. The surface Te lattice is indicated in (E and F). (H) Typical large-scale dI/dV spectra taken on different surface terraces (set point: Vb = 1 V, I = 100 pA).

Having identified the interface structure, the electronic states are studied by tunneling spectroscopy. Figure 1H shows large energy scale (±1 eV) dI/dV spectra taken on different surfaces. On Fe(Te, Se) substrate and 1-UC Bi2Te3, the line shapes of the spectra are similar to previous studies (, ). For 1-UC MnTe on Bi2Te3/Fe(Te, Se), the dI/dV displays a large bandgap about 500 meV wide, which is consistent with the idea that bulk MnTe is an insulator. However, the EF of 1-UC MnTe locates close to the upper edge of the gap. This is likely due to electron doping effect induced by interfacial charge transfer. Figure 2 (A and B) shows the quasiparticle interference (QPI) measurement on 1-UC MnTe, which further revealed that a shallow electron pocket crossed E, with a band bottom at −60(±10) meV (see fig. S3 for additional QPI data). This indicates that the electrons in 1-UC MnTe will participate in superconductivity (QPI patterns are also observed on 1-UC Bi2Te3 but displays a different dispersion; see fig. S4). As for the 2-UC MnTe, there is still a flat insulating gap, but the EF is at the gap center, indicating a weakened doping effect.

Fig. 2.

The tunneling spectra and QPI of MnTe/Bi2Te3/Fe(Te, Se) heterostructure.

(A) dI/dV map and its fast Fourier transform (FFT) image taken on 1-UC MnTe surface (Vb = 0.4 V, I = 100 pA). (B) Summary of the FFT line profiles taken at various energies, which shows an electron-like dispersion with a band bottom at Eb ≈ −60 meV. (C) Typical superconducting gap spectra taken on different surface layers. The positions of the multiple coherence peaks are marked by arrows and tracked by dashed lines. (D) A series of spectra taken across a boundary between 1-UC MnTe and 1-UC Bi2Te3 (inset: topographic image of the boundary, the positions where the spectra were taken are marked). (E) Temperature dependence of the proximity superconducting gap on 1-UC MnTe. The dashed curves are convolutions of the spectrum taken at T = 0.4 K with Fermi-Dirac distribution function at corresponding temperatures.

The low-energy dI/dV spectra that reflect superconducting state are then measured at T = 0.4 K on different surfaces, as summarized in Fig. 2C. The superconducting spectrum of Fe(Te, Se) substrate is fully gapped with multiple pairs of coherence peaks at ±1.5, ±1.9, and ±2.5 meV, respectively. We note that similar gaps were also reported in previous STM study and were assigned to different bands of Fe(Te, Se) (). The gap spectrum of 1-UC Bi2Te3 share similarities with Fe(Te, Se): Its major coherence peaks are at ±1.9 meV, while kink features at ±1.5 and ±2.5 meV can also be observed. This indicates a good superconducting proximity effect in Bi2Te3. However, for 1-UC MnTe, the spectrum displays rather unconventional behavior. First, in about 50% regions, a large “U”-shaped gap with a size of 2.9 meV (defined by the lowest kink feature at gap edge) is observed, as the one shown in Fig. 2C (red curve). Figure 2D displays how the gap evolves at a boundary between 1-UC Bi2Te3 and 1-UC MnTe, in which one can clearly see the gap enhancement in 1-UC MnTe region. Figure 2E shows the temperature dependence of the 1-UC MnTe gap. It almost disappeared above 13.6 K, which cannot be ascribed to the thermo broadening only (by comparing with numerically broadened spectra in Fig. 2E). This indicates that the gap of MnTe is still a proximity gap induced by Fe(Te, Se) (similar temperature dependence was also observed for the gap of 1-UC Bi2Te3; see fig. S5). Moreover, we have also observed vortex state in the gapped region of 1-UC MnTe (fig. S6), which further confirms its superconducting nature. The enlarged gap size in 1-UC MnTe is quite unexpected as one usually observes reduced gap in magnet/superconductor interfaces. Besides the enhanced gap, in other regions of 1-UC MnTe, we observed multiple in-gap states and zero-energy modes, which will be discussed in detail below. As for the 2-UC MnTe (green curve in Fig. 2C), in contrast to 1-UC MnTe, it shows a uniform gap similar to Bi2Te3, with pronounced coherence peaks located at ±1.9 meV.

In-gap states and zero-energy modes in 1-UC MnTe/Bi2Te3/Fe(Te, Se)

To further investigate the spatial distributions of superconducting states, Fig. 3 (A and B) shows a surface topography and its zero-bias dI/dV map (taken at B = 0 T), respectively. This area contains different terraces including 1-UC MnTe, 1-UC Bi2Te3, and Fe(Te, Se) substrate. The zero-bias conductance (ZBC) in the 1-UC Bi2Te3 and Fe(Te, Se) regions are all near zero, indicating a spatially uniform superconducting gap. However, on the 1-UC MnTe surfaces, there are many separated bright regions with finite ZBC. These small regions have a typical scale of 4 to 10 nm, and we refer to them as quasiparticle “puddles” below. Panels C and D of Figure 3 show two series of dI/dV spectra taken across two puddles indicated in Fig. 3B. A clear ZECP together with multiple discrete low-energy states are seen. Their intensities decay as leaving the puddle, but the energies do not change obviously. Figure 3 (E to N) shows the typical spectra taken in other 10 different puddles (spots 1 to 10 in Fig. 3B). In-gap bound states and ZECPs are commonly observed. Meanwhile, the spectra taken outside puddles show a clean U-shaped gap with a size of ≈2.9 meV (spots 11 to 13; Fig. 3O). We have repeatedly observed the quasiparticle puddles with a ZECP in different 1-UC MnTe terraces (an additional dataset is shown in fig. S7). By fitting the dI/dV spectra with Gaussian peaks, we summarized the appearance probability of the in-gap states at different energies in Fig. 3P, the ZECP has a probability over 80%. The peak width [full width at half maximum (FWHM)] of the ZECP is in the range of 0.35 to 0.55 meV, which is close to the energy resolution of our system.

Fig. 3.

ZECP and in-gap states in 1-UC MnTe/Bi2Te3/Fe(Te, Se).

(A) STM image of the heterostructure (80 nm by 60 nm). (B) Zero-bias dI/dV map of the area in (A) at B = 0 T (set point: Vb = 10 mV, I = 100 pA). The boundary of 1-UC MnTe terrace is tracked by the white dashed curve. (C and D) dI/dV spectra taken along the red arrows A and B in (B), respectively (set point: Vb = 3 mV, I = 200 pA). (E to O) dI/dV spectra measured on numbered spots in (B). The red solid curves in each panel are Gaussian peak fitting (dashed lines are individual peaks). (P) Statistics of the appearance probability of the in-gap states at different energies.

The quasiparticle puddles appear to distribute randomly. We have checked that the ZECP/in-gap state is not directly induced by observed defects in 1-UC MnTe (see fig. S2), and the distribution of quasiparticle puddles are also not correlated to surface defects or unevenness (see section S7 and fig. S8). This indicates that these in-gap states are unlikely the Yu-Shiba-Rusinov (YSR) state induced by impurities in MnTe. We also checked that the defects in exposed 1-UC Bi2Te3 do not induce YSR state either (fig. S2), which excludes that the in-gap states in 1-UC MnTe are induced by the defects of the underlying Bi2Te3 layer.

To explore the origin of ZECP and the in-gap states, we further measured their response to external magnetic field, elevated temperature and lowered tunneling barrier. Figure 4A shows the typical spectra measured under out-of-plane field. The ZECP persists and does not show splitting behavior up to B = 10 T (detailed analysis show that the ZECP is only slightly broadened by ~0.14 meV at B = 10 T, but substantial shift or broadening is observed for the non–zero-energy state; see section S8 and fig. S9). Meanwhile, the shape and spatial distribution of the quasiparticle puddles do not show obvious change either (fig. S10). This robustness against out-of-plane B field for ZECP also disfavors YSR-like states, which should split with a Zeeman energy (~1.1 meV for B = 10 T). Figure 4B shows a temperature dependence of the in-gap states spectra. All the in-gap states are broadened quickly at elevated temperatures and disappear near the TC of Fe(Te, Se), which indicates that they are still quasiparticle states related to superconductivity.

Fig. 4.

Magnetic field, temperature, and tunneling transmission dependence of the in-gap states and ZECP.

(A) dI/dV spectra taken at the same quasiparticle puddle in 1-UC MnTe under different magnetic field (Vb = 10 mV, I = 200 pA). (B) Temperature dependence of the dI/dV spectra with in-gap states (dashed curves are convolutions of the T = 0.4 K spectrum with the Fermi-Dirac distribution at elevated temperatures). (C) Evolution of dI/dV spectra with increasing tunneling transmission GN (GN = Iset/Vb, Vb = 2 mV). (D to F) Representative dI/dV spectra taken at different GN, and the Gaussian fit to the three low-energy peaks (referred by E±1, E0). (G) The tunneling conductance at 0 and −1.9 mV as a function of GN. The absolute conductance value is calibrated by numerical differential of the I/V curve. (H and I) Energy positions and peak width (FWHM) of three fitted peaks (E0, E±1) as a function of GN, respectively.

It was shown that the tunneling spectra at low barrier can help clarify the nature of bound states in superconductors, e.g., a Majorana mode should have quantized conductance of 2e2/h at strong coupling region and sufficiently low temperature (–). We measured the dI/dV spectra at reduced tip-sample distance via gradually increasing Iset at fixed Vb of 2 meV (GN = Iset/Vb describes the tunneling transmission). The results are summarized in Fig. 4C. As GN increases, the ZECP maintains its position, while the side peaks gradually move to higher energy (tracked by dashed lines). The three representative spectra shown in Fig. 4 (D to F) that are taken at different GN clearly display such behavior. Furthermore, when the conductance of ZECP reached about 0.22 (2e2/h), it starts to saturate, but the side peaks keep increasing and exceed the ZECP. In Fig. 4G, we plot the ZBC as a function of GN, and the saturation is directly evidenced. A similar behavior is also observed for other ZECPs (see fig. S11 for another dataset). We note that this behavior resembles the vortex zero modes reported in Fe(Te, Se), whose conductance are also saturated below 2e2/h (). It could be due to temperature broadening on a quantized peak, as the temperature here (T = 0.4 K) is similar to the case in ().

To see detailed evolution of the in-gap states with increasing GN, we performed Gaussian fit to the ZECP (E0) and the two lowest side peaks (refer as E±1). The fitted peak positions and peak width (FWHM) as a function of GN are summarized in Fig. 4 (H and I, respectively). The FWHM of E0 and E±1 states all increase with GN, and they nearly have similar values, which indicates that all these states have similar tunneling broadening at high GN (). Thus, the ZECP is distinguished from the E±1 peaks for its nonshifting behavior. We notice that topologically trivial YSR states were reported to have energy shift when the tip-sample distance changes (), which was attributed to tip-induced gating effect. In our heterostructure, we also occasionally observed these YSR-like shifting peaks (shown in fig. S12). Since the tunneling spectroscopy of 1-UC MnTe (Figs. 1H and 2, A and B) evidenced a bandgap and a shallow electron pocket with kF ≈ 0.08 (1/Å), which corresponds to a low carrier density, the tip gating could also be the cause of the shifting peaks such as E±1. Therefore the nonshifting ZECP and its saturated conductance would suggest its nontrivial origin.

Absence of in-gap states and gap enhancement in 2-UC MnTe/Bi2Te3/Fe(Te, Se)

We also measured the superconducting state in 2-UC MnTe/Bi2Te3. Figure 5A shows a topographic image that contains both 1- and 2-UC MnTe terraces. The height of the second MnTe layer is similar to 1-UC MnTe (Fig. 5C). Figure 5B is the zero-bias dI/dV map taken on the surface of Fig. 5A. On the 1-UC MnTe region, there are similar quasiparticle puddles in which ZECPs are found (Fig. 5D, points 2 to 4). However, for the 2-UC MnTe region, no such in-gap states were observed. Its dI/dV spectra (Fig. 5E) display a uniform full superconducting gap with a similar shape to that of 1-UC Bi2Te3/Fe(Te, Se), which is also different from the enhanced gap observed in 1-UC MnTe nearby (red curve in Fig. 5E, taken on point 1 in Fig. 5B). As the large-scale spectrum of 2-UC MnTe has a wide insulating gap with EF in the middle (Fig. 1H), the low-energy superconducting gap observed in 2-UC MnTe is likely from direct tunneling into the underneath Bi2Te3/Fe(Te, Se). Thus, these findings indicate that the 2-UC MnTe layer does not affect superconductivity of the system at all.

Fig. 5.

STM characterization of 2-UC MnTe/Bi2Te3/Fe(Te, Se).

(A) Topological image including a 2-UC MnTe terrace (53 nm by 70 nm, Vb = 1 V, I = 20 pA). The white dashed curve indicates the edge of 2-UC MnTe terrace. (B) Zero-bias dI/dV map taken in the area of (A) (set point: Vb = 10 mV, I = 150 pA). (C) The height profile along the red dashed line in (A). (D) dI/dV spectra measured at the 1-UC MnTe regions that show a ZECP [on the spots 2 to 4 in (B)]. (E) A series of dI/dV taken along the green arrow in (B) (on 2-UC MnTe), and the spectrum taken on spot 1 in (B) (on 1-UC MnTe; set point: Vb = 10 mV, I = 150 pA)

Calculations on the magnetism of 1- and 2-UC MnTe/Bi2Te3

The quite different superconducting states in 1- and 2-UC MnTe imply that magnetism may play an important role. Since the bulk MnTe displays interlayer AFM and intralayer ferromagnetic state (, ), one may expect that the magnetism in 1- and 2-UC MnTe is intrinsically different. However, it is also known that magnetism could change substantially in thin film or at the interface. Therefore, we performed first-principle calculations on the magnetism of 1- and 2-UC MnTe/Bi2Te3 heterostructures (details are shown in section S11). First, sufficient structure optimization of 1- and 2-UC MnTe/Bi2Te3 were performed, and the results are shown in Fig. 6 (A to C). They display similar in-plane lattice constant and interlayer height as that measured by STM. Then, we explore their most stable magnetic structures and evaluate the Heisenberg exchange coupling (J) in the local spin density approximation plus U (LSDA+U) framework. We find that 1-UC MnTe/Bi2Te3 favors nearest-neighbor AFM coupling (AFM1 structure) and the magnetic moment of Mn lies in plane, while the 2-UC MnTe/Bi2Te3 favors an interplane AFM (AFM2) with out-of-plane oriented moments, as sketched in Fig. 6 (D and E). The magnetic anisotropy energy of both the AFM1 and AFM2 states are larger than 5 meV. Since MnTe has a triangular lattice, the nearest-neighbor AFM1 state will have intrinsic frustration.

Fig. 6.

Calculated magnetic structure of 1- and 2-UC MnTe on Bi2Te3.

(A and B) Side views of optimized structure of 1- and 2-UC MnTe/Bi2Te3 interface, respectively. The blue, purple, and yellow balls represent Mn, Bi, and Te atoms, respectively. (C) Top view of the interface. (D and E) Local magnetic structure of 1-UC MnTe/Bi2Te3 and 2-UC MnTe/Bi2Te3, respectively.

The calculated AFM1 state in 1-UC MnTe/Bi2Te3 is different from single UC MnTe in the bulk form. This is checked to be due to interfacial effect (e.g. lattice strain), as our calculation can correctly predict the in-plane AFM state of bulk MnTe (see the Supplementary Materials). However, the above LSDA+U calculations only involve Heisenberg interactions, while for magnetic systems without inversion symmetry (such as heterostructures), SOC should give rise to DMIs that favor twisted spin configuration. On the basis of a first-principles linear response (FPLR) approach, we further calculated the DMI together with Heisenberg parameters of 1-UC MnTe/Bi2Te3 (section S11). The Heisenberg coupling (J) estimated by FPLR agrees well with that calculated by LSDA+U (≈20.0 meV). Meanwhile, the strength of nearest-neighbor DMI ∣D∣ is estimated to be 4.2 meV (specific values of the D vector are shown in table S6). Such a large DMI is originated from the large SOC of Bi2Te3 and the inversion asymmetric interface. We notice that several recent theoretical works on the structural asymmetric Mn dichalcogenide also give very large DMI (, ).

DISCUSSION

The robust ZECPs with nontrivial properties and enhanced superconducting gap in 1-UC MnTe/Bi2Te3/Fe(Te, Se) are rather unconventional phenomena. Below, we will discuss their origin. First, the observed ZECPs are located inside of the MnTe terrace, not at the edge. This excludes their origin of chiral edge mode that was predicted for ferromagnet/TI/superconductor heterostructures (, ). A natural explanation is that the 1-UC MnTe here is not ferromagnetic, as shown by above calculations. Second, these randomly distributed ZECPs seem to resemble the tunneling spectra of previously reported ferromagnet/superconductor heterostructures (, ), in which ZECPs were also observed and were attributed to spin-triplet pairing state (, ). However, we notice that this spin-triplet state would require a spin rotation at the interface (i.e., misalignment between the interfacial spin and bulk magnetization of the ferromagnet layer) and therefore is usually sensitive to external magnetic field (, ). In our measurement, the ZECP is robust against magnetic field (at least for out-of-plane field), and the MnTe layer is only 1 UC thick, which is unlikely to give interfacial spin misalignment. Thus, the spin-triplet pairing scenario is also not favored here.

Then, the above calculations of magnetism can provide an alternative explanation. It was shown that DMI and frustrated AFM state can both promote (in-plane) noncolinear spin texture such as skyrmions (–). Here, the calculated DMI strength in 1-UC MnTe/Bi2Te3 is even larger than that of the Co/Pt (≈3.0 meV) and Fe/Ir (≈1.7 meV) thin films (, ), in which skyrmions were observed with a typical size down to a few nanometers. Recently, a number of theoretical works have shown that various types of noncolinear spin texture can induce unconventional pairings in magnet/superconducting interface, which often manifest themselves as in-gap states (–). Particularly, zero-energy modes can be induced inside of ferromagnetic skyrmions (, , ) or at the end of AFM skyrmion chains under certain conditions (). Here, the ZECPs/in-gap states in 1-UC MnTe/Bi2Te3/Fe(Te, Se) are distributed in separated regions with a typical size of 4 to 10 nm (Fig. 3 and fig. S7). As defect-induced YSR states are not the origin, they are likely induced by certain noncolinear spin textures residing in such length scale. Moreover, the in-plane oriented magnetic moments in AFM1 state with strong anisotropy energy (>5 meV) suggest that they should be robust against out-of-plane field, which is consistent with our measurement in Fig. 4A. For 2-UC MnTe/Bi2Te3/Fe(Te, Se), the interplane AFM2 state with out-of-plane oriented moments is not frustrated and its interfacial DMI will also be weaker. Therefore, one expects to see less affected superconductivity in 2-UC MnTe.

Besides, the enhanced superconducting gap in 1-UC MnTe is another remarkable founding. To our knowledge, this is a very rare case that a proximity induced gap in magnetic material can be notably larger than the host superconductor. Most tunneling measurements on magnet/superconductor heterostructures have reported a reduced gap (, –). Here, since a shallow electron pocket crossed EF in 1-UC MnTe (Fig. 2B), it indicates that the conducting electrons in 1-UC MnTe participated in superconductivity. Thus, our results indicate that interfacial 1-UC MnTe layer is a “doped AFM insulator.” We notice that the electron correlation in MnTe (in which the Mn atoms have 3d5 configuration) was reported to be strong (). The gap enhancement mechanism could be related to the frustrated magnetism and electron correlations in 1-UC MnTe, which is quite interesting and worthy of further investigation. As for the 2-UC MnTe, since its EF is well inside the insulating gap and the magnetism is also different, the gap enhancement is absent.

Last, we shall point out another possibility for observing in-gap states at B = 0 T: the “spontaneous” vortex state that was suggested for certain ferromagnet/superconductor interface () or superconductors with magnetic impurities (, , ). If assuming 1-UC MnTe only provides an effective magnetic field to the underneath Bi2Te3/Fe(Te, Se) and generated the in-gap state/ZECP, then one would expect to see similar ZECP in vortex of 1-UC Bi2Te3 by applying an external field. We have checked the vortex states of 1-UC Bi2Te3/Fe(Te, Se) (fig. S6), which show that there is a broad peak near zero bias at vortex center, and it gradually splits as leaving the vortex. This behavior is similar to that previously observed in Bi2Te3/Fe(Te, Se) (, ) and Bi2Te3/NbSe2 systems () but very different from the multiple in-gap states and nonsplitting ZECP in 1-UC MnTe here. Therefore, the spontaneous vortex state scenario is also not favored.

In summary, we have successfully fabricated a trilayer-type magnet-superconductor heterostructure of MnTe/Bi2Te3/Fe(Te, Se) that has atomically sharp interface and in which strong SOC is intentionally introduced. The observation of robust zero-energy modes and enhanced superconducting gap in 1-UC MnTe region strongly suggest an unconventional pairing state. Via first-principle calculations, we show that 1-UC MnTe/Bi2Te3 has large DMI due to its inversion asymmetric structure and strong SOC, which can generate noncolinear spin textures and give rise to ZECP. The enhanced gap could also be related to the frustrated magnetism and electron correlations in 1-UC MnTe. Therefore, this heterostructure provides a promising platform for further exploring unconventional and topological nontrivial superconductivity.

MATERIALS AND METHODS

Growth of the heterostructure

The sample growth was conducted in a low-temperature STM system (UNISOKU 1300) equipped with an MBE chamber. The FeTeSe1− (x ≈ 0.55) single crystal with TC = 14.5 K was cleaved in a UHV chamber at 78 K. Bi2Te3 (1 UC) was first grown on cleaved surface by co-depositing Bi (99.997%) and Te (99.999%) with a flux ratio of ≈1:10. Then, 1- to 2-UC MnTe was grown by co-depositing Mn (99.95%) and Te (99.999%) with a flux ratio of ≈1:10. The substrate was kept at 200°C during the growth and was post-annealed at the same temperature for 10 min to further improve the quality. The sample was transferred in situ from the MBE chamber to STM chamber after growth.

STM measurement

STM experiments were conducted at T = 0.4 or 4.2 K. PrIr tip was used after treatment on the Au(111) surface. The dI/dV spectra were collected by a standard lock-in technique with a modulation frequency of 741 Hz and amplitudes (ΔV) of 0.5 mV at 4.2 K and 0.1 mV at 0.4 K. The energy resolution and the zero point of sample bias were calibrated before measurement (fig. S1).