Symbolic regression is an interpretable machine learning method that learns mathematical expressions from given data. It naturally combines with Bayesian Inference which lets experts express their knowledge as prior distributions over equations. However, the infinite search space of mathematical expressions renders exhaustive search impractical, and Bayesian Inference remains costly. Therefore, we propose to execute the Bayesian Reasoning in the learned latent space of a trained Variational Autoencoder (VAE) and thereby exploit inherent structures in the search space. While latent spaces have been used to structure search spaces, our approach provides the probability of each mathematical expression rather than selecting the best one. We suggest practical approximations to the posterior distribution in latent space and obtain formula examples by sampling from the posterior using the Gaussian Process Hamiltonian Monte Carlo (GP-HMC) method. We have validated our method using various Koza, Nguyen, and self-generated datasets and compared it against genetic programming and SInDy concerning the Root Mean Square Error (RMSE). Keywords: Symbolic Regression, latent space, Variational Autoencoder, Character Variational Autoencoder, Grammar Variational Autoencoder, Bayesian Reasoning, Gaussian Process, Hamiltonian Monte Carlo, Gaussian Process Hamiltonian Monte Carlo