Real-time optimal quantum control for atomic magnetometers with decoherence

Quantum entanglement, in the form of spin squeezing, is known to improve the sensitivity of atomic sensors to static or slowly varying fields. Sensing transient events presents a distinct challenge, requires different analysis tools, and has not been shown to benefit from entanglement in practically important scenarios such as spin-precession magnetometry. To address this, we apply concepts from continuous quantum measurements and estimation theory to optical atomic magnetometers, aiming to accurately model these devices, interpret their measurement data, control their dynamics, and achieve optimal sensitivity. Quantifying this optimal performance requires determining a fundamental quantum limit on sensitivity. We derive this limit, imposed by noise, and show that it scales at best linearly with sensing time and atom number N, ruling out any super-classical scaling. This limit is independent of the initial state, measurement, estimator, and measurement-based feedback, and depends only on the decoherence model and the strength of field fluctuations. Thus, finding an estimator that attains this bound proves the sensing strategy optimal. To approach this limit, we develop a quantum dynamical model scalable with N, based on a co-moving Gaussian approximation of the stochastic master equation, which includes measurement backaction and decoherence. This enables a real-time estimation and control architecture integrating an extended Kalman filter with a linear quadratic regulator. Simulating the magnetometer with our model and EKF+LQR strategy shows that quantum-limited tracking of constant and fluctuating fields is within reach of current atomic magnetometers. Our sensing strategy can also track biologically relevant signals, such as heartbeat-like waveforms, and drive the atomic ensemble into an entangled state, even when the measurement record is used for feedback but later discarded.