InfiniBand

Creator

Creator

Created

Created

2024 Mar 15 16:50

Editor

Editor

Edited

Edited

2025 Mar 6 23:22

Refs

Refs

IB, NVIDIA Quantum-2 InfiniBan

Stuck Process and SIGTERM Signal Interruption During Training with accelerate launch

Updated 2024 Feb 3 16:48



export NCCL_P2P_LEVEL=NVL
export NCCL_P2P_DISABLE=1
export NCCL_IB_DISABLE=1

Environment Variables — NCCL 2.20.3 documentation

NCCL has an extensive set of environment variables to tune for specific usage.

https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html

Recommendations

///////