Analyzing galaxy surveys at the level of three-dimensional density fields promises not only optimal cosmological constraints but also new insights into local dark matter structures and their evolution. Realizing this potential requires a forward model that links primordial density perturbations to the observed galaxy distribution. Models based on perturbation theory are particularly attractive for their computational efficiency and robustness to theoretical uncertainties. By basing these models on the Field Theory of Large Scale Structure, we can effectively marginalize over uncertainties in the bias relation between dark matter and galaxies. In this talk, I will present the LEFTfield code, discuss its description of the gravitational evolution and bias, and its speed and accuracy. On the road to including the full complexity of survey data, many observational effects need to be taken into account. I will discuss redshift-space distortions as one prominent example and give a outlook on the future challenges ahead.