Seonglae ChoA technique that treats gradients as comprising low- and high-frequency components and amplifies the slowly changing components to accelerate the onset of generalization.
Accelerated Grokking by Amplifying Slow Gradients
Grokfast: Accelerated Grokking by Amplifying Slow Gradients
One puzzling artifact in machine learning dubbed grokking is where delayed generalization is achieved tenfolds of iterations after near perfect overfitting to the training data. Focusing on the...
https://arxiv.org/abs/2405.20233
