Advances in deep neural networks in recent years have made them an attractive tool for ab-initio electronic structure calculations. In the first part of the talk, we will discuss advances in the theory of equivariant neural networks, which have found applications in graph neural networks, machine learning of molecular force fields, and ab-initio electronic structure. In the second part of the talk, we will discuss methods for second-order optimization of neural networks. While many of these advances have struggled to gain traction in mainstream machine learning, they are particularly well suited for variational quantum Monte Carlo calculations, and closely related to methods like stochastic reconfiguration. Lastly, we will discuss ways that these ideas have been applied to variational QMC calculations in recent years, including Neural Network Quantum States, the Fermionic Neural Network, and PauliNet.