Huggingface accelerate config

Creator

Creator

Seonglae Cho

Created

Created

2024 Mar 4 8:46

Editor

Editor

Seonglae Cho

Edited

Edited

2024 Mar 17 6:37

Refs

Refs

multi node training

accelerate config in the terminal of both machines and follow the prompts. after that, accelerate launch somthing.py as well

Deepspeed

FSDP

TorchDynamo

Accelerate Multi-GPU on several Nodes How to

Hi, I wonder how to setup Accelerate or possibly train a model if I have 2 physical machines sitting in the same network. Each machine has 4 GPUs. Can I use Accelerate + DeepSpeed to train a model with this configuration ? Can’t seem to be able to find any writeups or example how to perform the “accelerate config”. Thanks.

Accelerate Multi-GPU on several Nodes How to

https://discuss.huggingface.co/t/accelerate-multi-gpu-on-several-nodes-how-to/10736

Accelerate Multi-GPU on several Nodes How to

Recommendations

//////////