모멘텀 업데이트에 orthogonalization 적용 SOAP, Muon, and Beyond: Pushing LLM Pretraining ScalesHigher-order optimizers such as Muon and SOAP offer faster convergence than AdamW, but their computational cost and numerical stability challenges have limited adoption at scale. In this work, we...https://arxiv.org/abs/2607.20548Muon: An optimizer for hidden layers in neural networks | Keller Jordan blogMuon is an optimizer for the hidden layers in neural networks. It is used in the current training speed records for both NanoGPT and CIFAR-10 speedrunning. Many empirical results using Muon have already been posted, so this writeup will focus mainly on Muon’s design. First we will define Muon and provide an overview of the empirical results it has achieved so far. Then we will discuss its design in full detail, including connections to prior research and our best understanding of why it works.https://kellerjordan.github.io/posts/muon/Kimi K Muon is Scalable for LLM TrainingRecently, the Muon optimizer based on matrix orthogonalization has demonstrated strong results in training small-scale language models, but the scalability to larger models has not been proven. We...https://arxiv.org/abs/2502.16982
Seonglae ChoSeonglae ChoSeonglae ChoSeonglae ChoSeonglae ChoSeonglae Cho
