Designing optimization algorithms manually is a laborious process. In Addition, many optimization algorithms rely on hand-crafted heuristics and perform poorly in applications for which they are not specifically designed. Thus, automating the algorithm design process is very appealing. Moreover, learned algorithms minimize the amount of a priori assumptions and do not rely on hyperparameters after training. Several works exist that present methods to learn an optimization algorithm. In this project, we focus on the reinforcement learning perspective. Therefore, any particular optimization algorithm is represented as a policy. Evaluation of the existing methods shows, learned algorithms outperform existing algorithms in terms of convergence speed and final objective value on particular training tasks. However, the inner mechanisms of learned algorithms largely remain a mystery. A first work has discovered that learned first-order algorithms show a set of intuitive mechanisms that are tuned to the training task. We aim to explore the inner workings of learned zeroth-order algorithms and compare our discoveries to previous works. To address this issue, we study properties of learned zeroth-order algorithms to understand the relationship between what is learned and the quantitative and qualitative properties, e.g., curvature or convexity of the objective function. Furthermore, we study the generalization in relation to these properties. Moreover, we explore the feasibility of finetuning a learned zeroth-order optimization algorithm to a related objective function. Finally we provide guidelines for training and application of learned zeroth-order optimization algorithms.