A long-standing problem in QFT and quantum gravity is the construction of an “IR-finite" S-matrix. In the gravitational case, the existence of these “infrared divergences” is intimately tied to the “memory effect” (i.e. the permanent displacement of test masses due to the passage of a gravitational wave). In this talk, I shall explain the origin of these connections and illustrate that the construction of an IR-finite S-matrix requires the inclusion of states with memory (which do not lie in the standard Fock space). In massive QED an elegant solution to this problem was provided by Faddeev and Kulish who constructed an incoming/outgoing Hilbert space of charged particles “dressed” with memory. We illustrate the ``preferred status'' of such states. However, we show that this construction fails in the case of massless QED, Yang-Mills theories, linearized quantum gravity with massless/massive sources, and in full quantum gravity. In the case of quantum gravity, we prove that the only "Faddeev-Kulish" state is the vacuum state. We also show that “non-Faddeev-Kulish” representations are also unsatisfactory. Thus, in general, it appears there is no preferred Hilbert space for scattering in QFT and quantum gravity. We argue that, at a fundamental level, one must formulate scattering theory without an a priori choice of Hilbert space. We outline the framework of such a manifestly IR-finite scattering theory. This talk is based on arXiv:2203.14334