Adam Optimizer

Creator

Creator

Created

Created

2023 Jun 18 8:47

Editor

Editor

Edited

Edited

2024 Dec 4 22:32

Refs

Refs

RMSprop + Momentum (
Adagrad)

파라미터마다 두 개의 추가 벡터를 유지하기 때문에 메모리 사용량이 크게 증가

First momentum

Second momentum

https://www.union.ai/blog-post/fine-tune-llama-2-with-limited-resources

https://moon-walker.medium.com/large-model-학습의-game-changer-ms의-deepspeed-zero-1-2-3-그리고-zero-infinity-74c9640190de

https://moon-walker.medium.com/large-model-학습의-game-changer-ms의-deepspeed-zero-1-2-3-그리고-zero-infinity-74c9640190de

\theta_{t+1} = \theta - \frac{\eta}{\sqrt{\hat{v}} + \epsilon} \hat{m_t}

$\eta$ means learning rate

$\epsilon$ to prevent 0 parent

https://arxiv.org/pdf/1412.6980.pdf

Recommendations

///////