Ultrahigh mobility and Rashba spin splitting in Sb-substituted bismuth telluride and bismuth selenide.

Topological insulators (TIs) such as Sb-doped BiTe and BiSe exhibit promising phenomena for advanced spintronics. While previous studies explored isolated doping levels; a systematic understanding of how Sb concentration influences topological behavior, Rashba-type spin splitting, and surface state formation is lacking. Here, we use density functional theory to investigate the structural, electronic, topological and transport properties of (BiSb)Te and (BiSb)Se thin films across 0 ≤ ≤ 1. We identify pronounced Rashba spin splitting in BiTe at = 0.5, 0.6, and 0.9 with in-plane helical spin textures. We identified the orbital origins of topological surface states and demonstrate that band inversion persists across the Sb doping range. At = 0.2, 0.4, and 0.8, calculated surface electron mobilities are consistent with experiments and increase an order of magnitude over SbTe, with minimal impact on bulk mobilities. These insights advance our understanding of TIs for spintronic and quantum device applications.