Horovod

Creator

Creator

Alan Jo

Created

Created

2020 Mar 28 15:56

Editor

Editor

Alan Jo

Edited

Edited

2024 May 5 8:56

Refs

Refs

Parallel Training

Uber에서 만든 Multi-GPU 및 분산 학습을 손쉽고 최적화된 형태로 진행할수 있게 만들어주는 Framework

TensorFlow뿐만 아니라 Keras, PyTorch, MXNet도 백엔드로 지원한다. Uber측에서 실험한 기본 Distributed TensorFlow를 사용할 때와 Horovod를 사용할때의 Benchmark 성능차이는 아래와 같다

OpenMPI error for Tensor so good to use NCCL even for performance

ValueError('Neither MPI nor Gloo support has been built. )

Updated 2024 May 7 7:6

Build a Conda Environment with GPU Support for Horovod — Horovod documentation

In this section we describe how to build Conda environments for deep learning projects using Horovod to enable distributed training across multiple GPUs (either on the same node or spread across multuple nodes).

Build a Conda Environment with GPU Support for Horovod — Horovod documentation

https://horovod.readthedocs.io/en/stable/conda.html

Recommendations

//////