Training Memory

• weights

• gradients

• optimizer states

• activations

• KV cache

• Data Parallelism

• FSDP

• Distributed ML

• Mixed Precision

• Computational Graph

Frameworks

• Prime RL

• Slime

• Megatron