Seonglae ChoDouble descent of Generalization performance
In a broad definition, Intelligence can be defined by whether Deep double descent occurs or not. It goes beyond memorization to achieve generalization.
- First descent
- Second descent
2019
Deep double descent
We show that the double descent phenomenon occurs in CNNs, ResNets, and transformers: performance first improves, then gets worse, and then improves again with increasing model size, data size, or training time. This effect is often avoided through careful regularization. While this behavior appears to be fairly universal, we don’t yet fully understand why it happens, and view further study of this phenomenon as an important research direction.
https://openai.com/index/deep-double-descent/
Deep Double Descent: Where Bigger Models and More Data Hurt
We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better. Moreover, we show...
https://arxiv.org/abs/1912.02292
2026
Double descent happens at model capacity where peak is exactly where data size = capacity.
arxiv.org
https://arxiv.org/pdf/2505.24832