Seonglae ChoReverse of Catastrophic forgetting
- memorization
- in-distribution generalization
- cross-distribution generalization
Grokking Methods
From OpenAI and Google researchers examines how neural networks generalize on small, algorithmically generated datasets. A network significantly improves its generalization performance after a point of overfitting, achieving perfect generalization in certain cases. This study is significant as it delves into the understanding of generalization in overparameterized neural networks beyond just memorizing finite training datasets.
Neel Nanda ICLR 2023 with Mechanistic interpretability
To find "progress measures," a transformer model was trained on modular addition tasks where grokking was observed. Reverse-engineering the implemented algorithm revealed that the model maps inputs to rotations on a circle and corresponds addition to rotation, specifically using discrete Fourier transforms and trigonometric identities to perform addition operations.
- Restricted Loss: Loss when non-key frequencies are removed
- Excluded Loss: Loss when only key frequencies are removed
Generalization
- Memorization (memorizing training data)
- Circuit formation (forming generalizable algorithms internally)
- Cleanup (removing memorization mechanisms)
Acceleration method
2023
Analyzes the transition between “memorization” and “generalization” as dataset size increases, and shows that the “middle regime” previously mistaken for optimization failure can instead be explained by superposition of linear features (July 2023).
Explores how overfitting and memorization can be explained through the lens of superposition. The core claim is that when a neural network is trained on a finite dataset, memorization occurs because the network stores individual data points in superposition rather than learning generalizing features. During the transition between these two regimes, the Deep double descent phenomenon is observed.