Seonglae ChoOrdered - Vanishing Gradient
Chaotic - Exploding gradient
Edge of Chaos
Therefore, when performing Weight Initialization, setting ensures that gradients are stably propagated, latent representations are both expressive and stable, and the network reaches the critical learning regime (edge of chaos).
Exponential expressivity in deep neural networks through transient chaos
We combine Riemannian geometry with the mean field theory of high dimensional chaos to study the nature of signal propagation in generic, deep neural networks with random weights. Our results...
https://arxiv.org/abs/1606.05340
Mean Field Residual Networks: On the Edge of Chaos
Part of
Advances in Neural Information Processing Systems 30 (NIPS 2017)
https://papers.nips.cc/paper_files/paper/2017/hash/81c650caac28cdefce4de5ddc18befa0-Abstract.html
Activation