Seonglae ChoReverse of Catastrophic forgetting
- memorization
- in-distribution generalization
- cross-distribution generalization
Deep Learning presents a challenge to classical statistical learning theory. Neural networks often achieve zero training error, yet they generalize well to unseen data. This contradicts traditional expectations and makes many classical generalization bounds ineffective.
Sparse activation and the Superposition Hypothesis have been proposed as possible explanations for the Grokking phenomenon, where models learn to activate sparsely and generalize well after initially overfitting when trained on very large datasets.
From OpenAI and Google researchers examines how neural networks generalize on small, algorithmically generated datasets. A network significantly improves its generalization performance after a point of overfitting, achieving perfect generalization in certain cases. This study is significant as it delves into the understanding of generalization in overparameterized neural networks beyond just memorizing finite training datasets.
Neel Nanda ICLR 2023 with Mechanistic interpretability
To find "progress measures," a transformer model was trained on modular addition tasks where grokking was observed. Reverse-engineering the implemented algorithm revealed that the model maps inputs to rotations on a circle and corresponds addition to rotation, specifically using discrete Fourier transforms and trigonometric identities to perform addition operations.
- Restricted Loss: Loss when non-key frequencies are removed
- Excluded Loss: Loss when only key frequencies are removed
Generalization
- Memorization (memorizing training data)
- Circuit formation (forming generalizable algorithms internally)
- Cleanup (removing memorization mechanisms)
arxiv.org
https://arxiv.org/pdf/2301.05217
Emergent Abilities of Large Language Models
Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon...
https://arxiv.org/abs/2206.07682
A Mechanistic Interpretability Analysis of Grokking — LessWrong
A significantly updated version of this work is now on Arxiv and was published as a spotlight paper at ICLR 2023 …
https://www.lesswrong.com/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking
Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
In this paper we propose to study generalization of neural networks on small algorithmically generated datasets. In this setting, questions about data efficiency, memorization, generalization, and speed of learning can…
https://ar5iv.labs.arxiv.org/html/2201.02177
Acceleration method
Grokfast: Accelerated Grokking by Amplifying Slow Gradients
One puzzling artifact in machine learning dubbed grokking is where delayed generalization is achieved tenfolds of iterations after near perfect overfitting to the training data. Focusing on the...
https://arxiv.org/abs/2405.20233
MLP Interpretability
A paper explaining the internal mechanism of the Grokking phenomenon in small neural networks learning modular addition through Fourier features + lottery ticket structure + phase alignment process. What the model actually learns: when a two-layer neural network solves modular addition, each neuron learns a single-frequency Fourier feature. In other words, it solves the problem by transforming it into a periodic signal decomposition problem rather than arithmetic. Previous research only discovered that "neurons learn frequencies," but this paper explains how those features are combined into a complete algorithm and why generalization suddenly occurs. Modular addition is special because it can be completely expressed with Fourier bases, making it possible to precisely analyze the internal mechanism, which is why it was chosen as a toy model.
After the memorization phase, phase alignment aligns the frequencies' phases, causing the entire structure to operate like a single algorithm. Then grokking occurs with an explosion in generalization performance. In other words, Grokking is not about feature discovery but rather about alignment or composition of already-discovered features.
Similar to the Lottery Ticket Hypothesis, there already exists a subnetwork within the network that can implement the correct algorithm. Learning is the process of "activating" that structure.
arxiv.org
https://arxiv.org/pdf/2602.16849
2023
Analyzes the transition between “memorization” and “generalization” as dataset size increases, and shows that the “middle regime” previously mistaken for optimization failure can instead be explained by superposition of linear features (July 2023).
Circuits Updates - July 2023
We report a number of developing ideas on the Anthropic interpretability team, which might be of interest to researchers working actively in this space. Some of these are emerging strands of research where we expect to publish more on in the coming months. Others are minor points we wish to share, since we're unlikely to ever write a paper about them.
https://transformer-circuits.pub/2023/july-update/index.html
Explores how overfitting and memorization can be explained through the lens of superposition. The core claim is that when a neural network is trained on a finite dataset, memorization occurs because the network stores individual data points in superposition rather than learning generalizing features. During the transition between these two regimes, the Deep double descent phenomenon is observed.
Superposition, Memorization, and Double Descent
In a recent paper , we found that simple neural networks trained on toy tasks often exhibit a phenomenon called superposition , where they represent more features than they have neurons. Our investigation was limited to the infinite-data, underfitting regime. But there's reason to believe that understanding overfitting might be important if we want to succeed at mechanistic interpretability, and that superposition might be a central part of the story.
https://transformer-circuits.pub/2023/toy-double-descent/index.html