Huggingface MultiGPU Traininng

Creator

Creator

Seonglae Cho

Created

Created

2023 Nov 22 12:24

Editor

Editor

Seonglae Cho

Edited

Edited

2024 Mar 8 6:59

Refs

Refs

[Benchmark] HF Trainer on RTX-3090

Updated 2022 Jan 19 18:13

adamw_apex_fused for

AdamW Optimizer

Supervised Fine-tuning Trainer

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Supervised Fine-tuning Trainer

https://huggingface.co/docs/trl/sft_trainer#multi-gpu-training

Efficient Training on Multiple GPUs

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Efficient Training on Multiple GPUs

https://huggingface.co/docs/transformers/perf_train_gpu_many#scalability-strategy

Efficient Training on Multiple GPUs

Train with a script

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Train with a script

https://huggingface.co/docs/transformers/run_scripts#distributed-training-and-mixed-precision

Train with a script

Recommendations

////////