Machine Learning (ML) has gathered a lot of interest from industry and academia as it increases the range of tasks that can be performed by computers: speech recognition, sentiment analysis, cancer detection, and much more. Neural Networks (NN) are a particular branch of ML that has achieved great popularity after winning recent well known image recognition contests; however, they can take days or weeks to be trained, which directly impacts productivity of interested parties. This project aims to accelerate Parameter Servers (PS), a common distributed algorithm for increasing the scale of NN applications. This will be initially done by applying High Performance Computing (HPC) techniques on a PS using only one level of parallelism, the minibatch level, and later by also exploiting parallelism on gradient calculation. Both investigation lines will be supported by design (e.g. PCAM analysis) and modelling (e.g. computational models) techniques, programming models (e.g. MPI, CUDA), theoretical results (e.g. Amdahl's Law) and metrics (e.g. speedup, depth-complexity) from HPC. By accelerating NNs we expect to increase the pace at which research in the area is carried, as well as ease the industrial process of designing NN models and deploying them within applications or services.