Unit:
Κατεύθυνση Μεγάλα Δεδομένα και Τεχνητή ΝοημοσύνηΠληροφορική
Author:
Kostopoulou Kalliopi
Supervisors info:
Αλέξανδρος Ντούλας
Original Title:
Sparse Communication for Deep Learning
Translated title:
Sparse Communication for Deep Learning
Summary:
As the complexity of Neural Network architectures increases so does our need to develop better algorithmic solutions and infrastructures for distributed training. Data parallelism is a popular approach for distributing the workload of the training process to multiple workers. However, the gradient exchange that needs to take place between the workers requires extensive network communication which often causes a bottleneck. Compressed communication tackles this issue by reducing the volume of the communicated data. A wide range of gradient compression algorithms have been developed for this purpose and each one of them is usually followed by some properties regarding the network throughput, as well as the model’s ability to converge under this method. In this work, we step on various sparsification techniques and perform an even more aggressive reduction of the gradients’ size by applying several lossless or lossy encoding methods. More specifically, we employ ideas like curve fitting or the widely-known bloom filter data structures. While doing that we also develop a comprehensive framework that enables the integration of new experimental encoding methods.
Main subject category:
Technology - Computer science
Keywords:
compressed communication, deep learning, distributed systems
