Grokfast

A technique that treats gradients as comprising low- and high-frequency components and amplifies the slowly changing components to accelerate the onset of generalization.

Accelerated Grokking by Amplifying Slow Gradients

Grokfast: Accelerated Grokking by Amplifying Slow Gradients

One puzzling artifact in machine learning dubbed grokking is where delayed generalization is achieved tenfolds of iterations after near perfect overfitting to the training data. Focusing on the...

https://arxiv.org/abs/2405.20233

Grokfast

Recommendations