Control the variance to avoid softmax output would be like one hot vector
important normalization to have
Because k, q variance is much smaller than value
dividing by squared dimension size
Scaled Attention
Creator
Creator
Seonglae ChoCreated
Created
2024 Feb 28 3:55Editor
Editor
Seonglae ChoEdited
Edited
2024 Feb 28 4:0Refs
Refs
