We propose a general framework for solving machine learning based optimization problems via one-shot formulation. In particular, we are interested in the application to PDE-constrained optimization under uncertainty and inverse problems. In order to solve these problems typically a large number of PDEs need to be solved. In our approach, we replace the complex forward model by a physics informed surrogate model, e.g. a neural network, which is learned simultaneously when estimating the unknown parameters or solving the optimal control problem. We formulate a penalized empirical risk minimization problem and analyze its consistency in the large sample size limit. Moreover, we develop an algorithmic framework using a penalized stochastic gradient descent method in order to reduce the associated computational costs. This is joint work with Philipp Guth and Claudia Schillings.