Automatic parallelization deep learning middleware RaNNC wins First Place at PyTorch Annual Hackathon 2021

National Institute of Information and Communications Technology (NICT, Chairman: Hideyuki Tokuda) Data Driven Intelligent Systems Research Center (DIRECT) and National University of Tokyo (President: Teruo Fujii) Information Infrastructure Center jointly developed automatic parallelization deep learning Learning Middleware RaNNC (Rank, Rapid Neural Network Connector) was awarded the First Place (1st place, PyTorch Developer Tools & Libraries category) at PyTorch Annual Hackathon 2021.

PyTorch Annual Hackathon is the only event officially held by Facebook, the developer of PyTorch, as an event to compete for achievements related to PyTorch, a representative software for deep learning widely used all over the world, and many participants from all over the world. Are gathering. The award-winning middleware RaNNC dramatically simplifies learning large-scale neural networks, which was difficult with PyTorch's conventional functions. RaNNC is open source and open to the public, and you can download it for free, including for commercial purposes.

PyTorch Annual Hackathon is an event to develop software and machine learning models using PyTorch, a framework for describing neural networks used in deep learning. It is the only event officially held by Facebook, the developer of PyTorch, as an event to compete for achievements related to PyTorch, and has been held once a year since 2019. This year, 1,947 people from all over the world participated and received 65 applications.

RaNNC is middleware for automatically dividing a large-scale neural network and learning at high speed by parallel processing using multiple GPUs. When training a large-scale neural network, the training parameters do not fit in the GPU memory, so the neural network must be divided and processed in parallel on multiple GPUs. Therefore, in the past, it was necessary to manually describe the neural network in consideration of the division of the neural network and the speed of parallel processing, which required a lot of labor and specialized knowledge. On the other hand, RaNNC automatically automates the neural network so that it fits in the GPU memory and high learning speed can be obtained by parallel processing, given a neural network description that assumes the use of a single GPU and does not consider parallelization. Because it is divided by, it can greatly facilitate the learning of large-scale neural networks.

RaNNC has been open to the public since March 2021, but we have added new features when applying for the PyTorch Annual Hackathon 2021. Specifically, in order to reduce the GPU memory required for large-scale neural network learning, many of the learning parameters are placed in a larger main memory, and it is necessary at the timing when calculation using the GPU is required. We have realized the function to move only the data to the memory of the GPU. This makes it possible to train large-scale neural networks with smaller GPU memory.

RaNNC was developed as a joint research between the National Institute of Information and Communications Technology and the University of Tokyo. The recipients of the award are as follows.

So far, RaNNC has been used to realize automatic parallelization of learning of neural networks with 100 billion parameters. Conventionally, learning of a neural network of this scale has been realized by a professional engineer drastically modifying the description of the neural network originally described without considering parallelization, and the cost of the modification is high. It was a big one. RaNNC, on the other hand, automatically parallelizes the learning of neural networks written on the assumption that a single GPU is used without considering parallelization at all, and realizes high-speed learning. This eliminates the need to modify the description of the neural network for parallel processing, and enables parallel learning of large-scale neural networks at lower cost and in a shorter time.

In addition, while the main existing software for large-scale neural network learning can be applied only to specific types of neural networks such as Transformer, RaNNC is basically applicable to any type of neural network. there is.

RaNNC is published on GitHub along with source code and usage examples (URL: https://github.com/nict-wisdom/rannc). The license is the MIT license and can be used free of charge, including for commercial purposes.