Thermal and vacuum fluctuations of the electromagnetic field can be described through macroscopic quantum electrodynamics. These fluctuations are responsible for changes in the Casimir-Polder frequency shift and decay rate at finte and zero temperature, respectively. In this work, we investigate how the quantum friction experienced by a polarizable charged particle moving with constant velocity parallel to a planar interface is modified when the latter consists of a chiral media or nonreciprocal media, with special focus on topological insulators. We obtain the Casimir-Polder frequency shift and decay rate, which generalize the respective quantities to matter with time-reversal symmetry breaking which violates the Lorentz reciprocity principle. We focus on the vacuum state as a point of departure that can easily be generalized to finite temperature and illustrate our findings by examining the nonretarded and retarded limits for five examples: a perfectly conducting mirror, a perfectly reflecting nonreciprocal mirror, a three-dimensional topological insulator, a perfectly reflecting chiral mirror and an isotropic chiral medium. We find different power laws for all these materials. Interestingly, we find bridges between chirality and nonreciprocity through both the frequency shift and the decay rate that arise as a consequence of the magnetoelectric coupling.