Seonglae ChoUpdate weights with reading tokens
Basically, the starting point of back propagation is a cross-entropy loss that measures the difference between the softmax logits distribution calculated through the softmax function and the actual labels until the Positional Embedding and Token Embedding.
Techniques
- SGD/Adam is not as well behaved as other NN’s
- Require a lot of data and Larger batch sizes than usual with lower learning rates
- Data Shuffling is important
- At the start of training, traditional transfomers are unstable which is caused by unbalanced gradient. So we useLearning rate Warmup
- Unstable gradient is understood to be due to Residual Connection & Layer Normalization and potentially amplified by Adam
- We need Adaptive learning rate where is algorithmic step and is the number of warmup steps
- catering the learning rate to the largest gradient with warmup
- Regularization through dropout and label smoothing to prevent overfitting
Transformer Training Steps
Technical tips
Weight Initialization for transformer with Learning rate Warmup
Layer Normalization for transformer with Pre-Norm