Seonglae Cho2023
Analyzes the transition between “memorization” and “generalization” as dataset size increases, and shows that the “middle regime” previously mistaken for optimization failure can instead be explained by superposition of linear features (July 2023).
Circuits Updates - July 2023
We report a number of developing ideas on the Anthropic interpretability team, which might be of interest to researchers working actively in this space. Some of these are emerging strands of research where we expect to publish more on in the coming months. Others are minor points we wish to share, since we're unlikely to ever write a paper about them.
https://transformer-circuits.pub/2023/july-update/index.html
Explores how overfitting and memorization can be explained through the lens of superposition. The core claim is that when a neural network is trained on a finite dataset, memorization occurs because the network stores individual data points in superposition rather than learning generalizing features. During the transition between these two regimes, the Deep double descent phenomenon is observed.
Superposition, Memorization, and Double Descent
In a recent paper , we found that simple neural networks trained on toy tasks often exhibit a phenomenon called superposition , where they represent more features than they have neurons. Our investigation was limited to the infinite-data, underfitting regime. But there's reason to believe that understanding overfitting might be important if we want to succeed at mechanistic interpretability, and that superposition might be a central part of the story.
https://transformer-circuits.pub/2023/toy-double-descent/index.html
Double descent of Generalization performance
In a broad definition, Intelligence can be defined by whether Deep double descent occurs or not. It goes beyond memorization to achieve generalization.
- First descent
- Second descent
Deep Learning presents a challenge to classical statistical learning theory. Neural networks often achieve zero training error, yet they generalize well to unseen data. This contradicts traditional expectations and makes many classical generalization bounds ineffective.
Sparse activation and the Superposition Hypothesis have been proposed as possible explanations for the Grokking phenomenon, where models learn to activate sparsely and generalize well after initially overfitting when trained on very large datasets.
2019
Deep double descent
We show that the double descent phenomenon occurs in CNNs, ResNets, and transformers: performance first improves, then gets worse, and then improves again with increasing model size, data size, or training time. This effect is often avoided through careful regularization. While this behavior appears to be fairly universal, we don’t yet fully understand why it happens, and view further study of this phenomenon as an important research direction.
https://openai.com/index/deep-double-descent/
Deep Double Descent: Where Bigger Models and More Data Hurt
We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better. Moreover, we show...
https://arxiv.org/abs/1912.02292
2023
Analyzes the transition between “memorization” and “generalization” as dataset size increases, and shows that the “middle regime” previously mistaken for optimization failure can instead be explained by superposition of linear features (July 2023).
Circuits Updates - July 2023
We report a number of developing ideas on the Anthropic interpretability team, which might be of interest to researchers working actively in this space. Some of these are emerging strands of research where we expect to publish more on in the coming months. Others are minor points we wish to share, since we're unlikely to ever write a paper about them.
https://transformer-circuits.pub/2023/july-update/index.html
Explores how overfitting and memorization can be explained through the lens of superposition. The core claim is that when a neural network is trained on a finite dataset, memorization occurs because the network stores individual data points in superposition rather than learning generalizing features. During the transition between these two regimes, the Deep double descent phenomenon is observed.
Superposition, Memorization, and Double Descent
In a recent paper , we found that simple neural networks trained on toy tasks often exhibit a phenomenon called superposition , where they represent more features than they have neurons. Our investigation was limited to the infinite-data, underfitting regime. But there's reason to believe that understanding overfitting might be important if we want to succeed at mechanistic interpretability, and that superposition might be a central part of the story.
https://transformer-circuits.pub/2023/toy-double-descent/index.html