SPMD

Created

Created

2024 Feb 27 13:18

Editor

Editor

Seonglae Cho

Creator

Creator

Seonglae Cho

Edited

Edited

2024 Mar 17 5:49

Refs

Refs

Scalable Parallelization for ML Computation Graphs

To enable FSDP via SPMD

PyTorch on XLA Devices — PyTorch/XLA master documentation

PyTorch runs on XLA devices, like TPUs, with the torch_xla package. This document describes how to run your models on these devices.

PyTorch on XLA Devices — PyTorch/XLA master documentation

https://pytorch.org/xla/release/2.1/index.html#pytorch-xla-spmd-user-guide

https://arxiv.org/pdf/2105.04663.pdf

Recommendations

//////////