Huggingface accelerate FSDP

Created

Created

2024 Mar 8 6:56

Editor

Editor

Seonglae Cho

Creator

Creator

Seonglae Cho

Edited

Edited

2024 Mar 17 11:58

Refs

Refs

Best practice

Fine-tuning Llama 2 70B using PyTorch FSDP

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

https://huggingface.co/blog/ram-efficient-pytorch-fsdp

Fine-tuning Llama 2 70B using PyTorch FSDP

Accelerate Large Model Training using PyTorch Fully Sharded Data Parallel

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

https://huggingface.co/blog/pytorch-fsdp

Accelerate Large Model Training using PyTorch Fully Sharded Data Parallel

Usages

Fully Sharded Data Parallel

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

https://huggingface.co/docs/accelerate/usage_guides/fsdp#saving-and-loading

Fully Sharded Data Parallel

Fully Sharded Data Parallel

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

https://huggingface.co/docs/accelerate/usage_guides/fsdp

Fully Sharded Data Parallel

Recommendations

////////