By utilizing the quadratic dependency of the interferometry phase on time, the Hannover Very Long Baseline Atom Interferometer facility (VLBAI) aims for sub nm/s^2 gravity measurement sensitivity at 1s integration time. With its 10m vertical baseline, beyond applications in Earth observation VLBAI offers promising prospects in testing fundamental physics at the interface between quantum mechanics and general relativity. Here we discuss the challenges imposed on controlling the VLBAI facility's surroundings and report on the characterization of its magnetic, gravitational, and thermal environment. We show routes towards sub nm/s^2 accuracy in absolute gravimetry and parts in 10^13 and beyond for differential measurements.