Weight Initialization

Creator

Creator

Seonglae Cho

Created

Created

2023 Jun 6 8:40

Editor

Editor

Seonglae Cho

Edited

Edited

2025 Nov 11 22:40

Refs

Refs

Parameter initialization

weight to small random numbers

bias (zero or small nonzero)

Weight Initialization Usages

He initialization

Xavier Initialization

Gaussian Initialization

Kaiming initialization

0025 Initialization - Deepest Documentation

https://deepestdocs.readthedocs.io/en/latest/002_deep_learning_part_1/0025/

Backlinks

Mechanistic interpretability Transformer Training Randomized Transformer Meta Learning Edge of Chaos torch.manual_seed()Residual Connection AI Optimization

Recommendations

/////