Deep double descent

Creator

Creator

Seonglae Cho

Created

Created

2024 Sep 14 20:18

Editor

Editor

Seonglae Cho

Edited

Edited

2026 Jul 13 8:15

Refs

Refs

Bias-Variance Trade-off

In-context learning ability

Double descent of Generalization performance

In a broad definition,

Intelligence can be defined by whether

Deep double descent occurs or not. It goes beyond memorization to achieve generalization.

First descent

Overfitting

Second descent

https://openai.com/index/deep-double-descent/

2019

Deep double descent

We show that the double descent phenomenon occurs in CNNs, ResNets, and transformers: performance first improves, then gets worse, and then improves again with increasing model size, data size, or training time. This effect is often avoided through careful regularization. While this behavior appears to be fairly universal, we don’t yet fully understand why it happens, and view further study of this phenomenon as an important research direction.

https://openai.com/index/deep-double-descent/

Deep double descent

Deep Double Descent: Where Bigger Models and More Data Hurt

We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better. Moreover, we show...

https://arxiv.org/abs/1912.02292

2026

Double descent happens at model capacity where peak is exactly where data size = capacity.

https://arxiv.org/pdf/2505.24832

Backlinks

Transformer Model Overfitting Preetum Nakkiran AI Memory Capacity In-context learning ability Deep double descent Grokking Reversing Transformer Language Model

Recommendations

///////