The pursuit of energy-efficient deep neural network (DNN) is becoming increasingly important in the era of AI-driven applications. Computing resources and energy consumption are important factors for the increasing application of neural networks on resource-constrained devices such as smartwatches. This thesis explores methods and techniques for optimizing the energy efficiency of DNNs through pruning and neural architecture search (NAS). While current research claims that these methods lead to improved energy efficiency, there is little empirical evidence to support this claim. This work presents a thorough investigation of pruning algorithms and NAS techniques with the goal of reducing energy consumption without compromising model accuracy. Experimental evaluations on various baseline architectures and datasets, including MNIST, CIFAR-10, DeepSat, and California Housing, inform the challenges in achieving tangible improvements in energy efficiency. Key metrics such as model size, computation time, power consumption, energy consumption, and FLOPs are carefully measured and compared against different methods and datasets. Experiments will be conducted on NVIDIA GeForce RTX 3090 GPUs and other hardware configurations to gain comprehensive insights into energy consumption patterns. The results reveal a lack of significant changes in power consumption, indicating that power consumption is not affected by unstructured pruning methods alone. Furthermore, the results underline the importance of considering different objectives and metrics when optimizing energy efficiency for different tasks and applications of neural networks. To summarize, although the results do not meet the expectations raised by current research in this field, they provide valuable insights and point the way for future research towards sustainable and energy-efficient AI systems. With our main contribution to this work, the NASO framework, we provide software that allows us to extend our research to other hardware and pruning methods.