Seonglae ChoSOAP, Muon, and Beyond: Pushing LLM Pretraining Scales
Higher-order optimizers such as Muon and SOAP offer faster convergence than AdamW, but their computational cost and numerical stability challenges have limited adoption at scale. In this work, we...
https://arxiv.org/abs/2607.20548
Muon: An optimizer for hidden layers in neural networks | Keller Jordan blog
Muon is an optimizer for the hidden layers in neural networks. It is used in the current training speed records for both NanoGPT and CIFAR-10 speedrunning. Many empirical results using Muon have already been posted, so this writeup will focus mainly on Muon’s design. First we will define Muon and provide an overview of the empirical results it has achieved so far. Then we will discuss its design in full detail, including connections to prior research and our best understanding of why it works.
https://kellerjordan.github.io/posts/muon/
Muon is Scalable for LLM Training
Recently, the Muon optimizer based on matrix orthogonalization has demonstrated strong results in training small-scale language models, but the scalability to larger models has not been proven. We...
https://arxiv.org/abs/2502.16982