Gradient Clipping

Can be considered as an adaptive learning rate without smoothing

To prevent gradient explosion, values are clipped to not exceed a threshold. When FP overflow occurs, values are typically clipped according to their norm.

Trace anomaly

07-07 기울기 소실(Gradient Vanishing)과 폭주(Exploding)

깊은 인공 신경망을 학습하다보면 역전파 과정에서 입력층으로 갈 수록 기울기(Gradient)가 점차적으로 작아지는 현상이 발생할 수 있습니다. 입력층에 가까운 층들에서 가중치들이…

https://wikidocs.net/61375

Gradient Clipping

Can be considered as an adaptive learning rate without smoothing

Trace anomaly

Backlinks

Recommendations