Stable-to-unstable transition in quantum friction

We investigate the frictional force arising from quantum fluctuations when two dissipative metallic plates are set in a shear motion. While early studies showed that the electromagnetic fields in the quantum friction setup reach nonequilibrium steady states, yielding a time-independent force, other works have demonstrated the failure to attain steady states, leading to instability and time-varying friction under sufficiently low-loss conditions. Here, we develop a fully quantum-mechanical theory without perturbative approximations, and we unveil the transition from stable to unstable regimes of the quantum friction setup. Due to the relative motion of the plates, their electromagnetic response may be active in some conditions, resulting in optical gain. We prove that the standard fluctuation-dissipation leads to inconsistent results when applied to our system, and, in particular, it predicts a vanishing frictional force. Using a modified fluctuation-dissipation relation tailored for gain media, we calculate the frictional force in terms of the system Green's function, thereby recovering early works on quantum friction. Remarkably, we also find that the frictional force diverges to infinity as the relative velocity of the plates approaches a threshold. This threshold is determined by the damping strength and the distance between the metal surfaces. Beyond this critical velocity, the system exhibits instability, akin to the behavior of a laser cavity, where no steady state exists. In such a scenario, the frictional force escalates exponentially. Our findings pave the way for experimental exploration of the frictional force in proximity to this critical regime.