Since the invention of microprocessors the trend for improvement of CPU speed is mainly due to increasing the clock speed. After multi-core architecture introduction the focus has shifted toward increasing the number of cores. Now the trend is shifting from multi-core to many-core architecture. Modern GPUs are capable of general purpose calculation having hundreds of cores compared to CPU which have 2-12 cores. The GPU with large number of cores makes it another option for parallel computation and solving numerical problems. In 2007 Compute Unified Device Architecture (CUDA) launched by NVIDA makes it easier for programmers to implement parallel computation on GPU. The training algorithms for neural networks are potential computing applications which can be implemented on Single Instruction Multiple Data (SIMD) GPU architectures. The objective of this work is to analyze and illustrate the parallel implementation of Back Propagation training algorithm for feed forward Multilayer Perceptron (MLP) neural networks on NVIDIA GPUs i.e. Tesla S1060 and Tesla C2050 based on Fermi architecture. We also compare speedup between GPU and CPU models. The GPU model is implemented using CUDA parallel computing architecture whereas two CPU implementations are considered one with boost library as provided by HMI-Tec and the other implementation without boost library. The comparison between these models done with five standard benchmark data sets shows CUDA library based GPU model dominate up to factor of 12x against BLAS library based CPU implementation and 43.5x faster than standard C CPU implementation.