NCCL

NVIDIA Collective Communication Library

Windows are not supported

Cross-GPU tensor communication

Almost every CUDA based multi GPU training/inference server use NCCL

NCCL_SOCKET_IFNAME

NCCL_DEBUG

Using torch


python -c "import torch;print(torch.cuda.nccl.version())"

Install


wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
dpkg -i cuda-keyring_1.0-1_all.deb
apt update
apt install libnccl2=2.18.5-1+cuda12.1 libnccl-dev=2.18.5-1+cuda12.1

Install

Abstract

This NVIDIA Collective Communication Library (NCCL) Installation Guide provides a step-by-step instructions for downloading and installing NCCL.

https://docs.nvidia.com/deeplearning/nccl/install-guide/index.html

Env

Environment Variables — NCCL 2.20.3 documentation

NCCL has an extensive set of environment variables to tune for specific usage.

https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html

NVIDIA Collective Communications Library (NCCL)

The NVIDIA Collective Communication Library (NCCL) implements multi-GPU and multi-node communication primitives optimized for NVIDIA GPUs and Networking. NCCL provides routines such as all-gather, all-reduce, broadcast, reduce, reduce-scatter as well as point-to-point send and receive that are optimized to achieve high bandwidth and low latency over PCIe and NVLink high-speed interconnects within a node and over NVIDIA Mellanox Network across nodes.

https://developer.nvidia.com/nccl

NVIDIA Collective Communications Library (NCCL)

Operations — NCCL 2.6.4 documentation

Like MPI collective operations, NCCL collective operations have to be called for each rank (hence CUDA device) to form a complete collective operation. Failure to do so will result in other ranks waiting indefinitely.

https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/operations.html

NCCLX: Collective Communication for 100k+ GPUs

NCCLX extends NCCL with CTran, a host-driven transport supporting zero-copy, SM-free transfers, and lazy connection and channel allocation to reduce GPU resource use. Compared with NCCL, it delivers up to 11× faster training startup at 96,000 GPUs, reduces Llama 4 steady-state training step latency by up to 12%, and lowers Llama 4 Maverick end-to-end decoding latency by 15–80% across distributed configurations.

arxiv.org

https://arxiv.org/abs/2510.20171