Spectral-folding model and capacity boundary for orbital angular momentum mode sorting in Fabry–Pérot cavities

Fabry–Pérot (FP) cavities have been demonstrated as frequency-domain selection elements for orbital angular momentum (OAM) states of light, with applications in optical communications and photonic quantum information processing. However, a general analytical model for high-dimensional OAM mode sorting in FP cavities has been lacking. Here, we present an analytical model based on spectral folding in FP cavities. By mapping folded resonance positions onto a single free spectral range (FSR) circle, the cavity design problem is reduced to maximizing the minimum circular spacing between target modes. A unified performance criterion then links this spacing and the cavity finesse to practical sorting metrics such as sorting efficiency and extinction ratio, establishing a capacity boundary for arbitrary finite target sets under a given performance threshold. The bound is tight for uniform-step target sets, for which the optimal cavity geometry is derived in closed form; for general nonuniform sets, the design reduces to an exact finite search over a bounded set of rational candidates. To experimentally validate the model, we examine a representative boundary case involving nine consecutive OAM modes in the p = 0 Laguerre-Gaussian (LG) subspace. Experimental measurements confirm the predicted folded resonance ordering, where the next mode returns to the fundamental’s spectral neighborhood. For an equal-weight nine-mode superposition state, the sorting achieves an average efficiency of 93.19% and a mean extinction ratio of 11.41 dB across all target modes, closely matching theoretical predictions.