Edge of Chaos

Therefore, when performing

Weight Initialization, setting ensures that gradients are stably propagated, latent representations are both expressive and stable, and the network reaches the critical learning regime (edge of chaos).

Exponential expressivity in deep neural networks through transient chaos

We combine Riemannian geometry with the mean field theory of high dimensional chaos to study the nature of signal propagation in generic, deep neural networks with random weights. Our results...

https://arxiv.org/abs/1606.05340

arxiv.org

https://arxiv.org/pdf/1611.01232

Mean Field Residual Networks: On the Edge of Chaos

Part of Advances in Neural Information Processing Systems 30 (NIPS 2017)

https://papers.nips.cc/paper_files/paper/2017/hash/81c650caac28cdefce4de5ddc18befa0-Abstract.html

Activation

arxiv.org

https://arxiv.org/pdf/1902.06853

arxiv.org

https://arxiv.org/pdf/2107.09437

Edge of Chaos

Ordered -
Vanishing Gradient

Chaotic -
Exploding gradient

Edge of Chaos

Recommendations

Edge of Chaos

Ordered - Vanishing Gradient

Chaotic - Exploding gradient

Edge of Chaos

Recommendations

Ordered -
Vanishing Gradient

Chaotic -
Exploding gradient