Understanding the fundamental properties of complex systems with disordered microscopic interactions remains a longstanding challenge. Spin glass models provide a systematic framework for studying such systems. Notably, at low levels of thermal activation, these models exhibit a glassy phase characterized by broken ergodicity. The impact of small deviations from Hamiltonian dynamics has been widely debated. In this work, we explore how the breaking of interaction symmetry affect systems with marginally stable local equilibria. Our findings support the prevailing view that a true spin-glass phase cannot persist in the thermodynamic limit when interactions are asymmetric, to any degree. However, we demonstrate that large but finite systems can still show glass-like behavior, where a new form of athermal glassy dynamics is unleashed. Specifically, for the XY spin glass with infinite-range interactions, we compute the finite-size phase diagram.