torch.distributed.c10d

Creator

Creator

Seonglae Cho

Created

Created

2024 Mar 18 5:8

Editor

Editor

Seonglae Cho

Edited

Edited

2024 Mar 18 5:22

Refs

Refs

Pytorch Distribuited backend

Pytorch Rendezvous 와 달리 텐서통신 담당

backend

NCCL - multi-gpu multi-node training

Gloo - hieh compatibility like multi cpu training or single gpu training

MPI - high performance computing but need to separate pytorch compilation

notion image

Gloo에서도 all_gather 함수 일부분 지원 안하기때문에 사실상 nccl이 독점이라고 봐도 무방함

nccl에 port를 임의로 사용하는 문제가 있어서 사용이 까다롭다

Random port with firewall issue

torch.distributed server back-connects to clients on random ports leading to firewall problems

Updated 2023 Jun 9 18:58

[RFC] Change --standalone to bind to a random port

Updated 2023 Aug 26 4:20

Recommendations

/////////