Let $H_0$ be self-adjoint and let $A$ (playing the role of the annihilation operator) be $H_0$-bounded. Assuming some additional hypotheses on $A$ (so that the creation operator $A^*$ is a singular perturbation of $H_0$), by a variant of Krein's resolvent formula, we build self-adjoint realizations $H$ of the formal Hamiltonian $H_0+A^*+A$ with $D(H_0)\cap D(H)=\{0\}$. We give an explicit characterization of $D(H)$ and provide a formula for the resolvent difference $(-H+z)^{-1}-(-H_0+z)^{-1}$. Moreover, we consider the problem of the description of $H$ as a (norm resolvent) limit of sequences of the kind $H_0+A_n^*+A_n+E_n$, where the $A_n$'s are regularized operators approximating $A$ and the $E_n$'s are suitable renormalizing bounded operators. These results show the connection between the construction of singular perturbations of self-adjoint operators by Krein's resolvent formula and nonperturbative theory of renormalizable models in Quantum Field Theory; in particular, as an explicit example, we consider the Nelson model.