new improvement than Post-Norm
Stable since the derivation term is simpler than Post-Norm without additional multiplication
And this Constant gradients enable larger learning rates
Pre-Norm
Creator
Creator
Seonglae ChoCreated
Created
2024 Nov 25 15:31Editor
Editor
Seonglae ChoEdited
Edited
2024 Nov 25 15:37Refs
Refs