Seonglae ChoMassive Activation
Special token, delimiter, conjunction, preposition, first token, number token, weak semantics
This massive activation functions like a constant bias within the model, where setting it to 0 degrades performance while maintaining performance when fixing to mean value. During LayerNorm/RMSNorm, large values dominate the variance, causing the token's normalization vector to become distinctive and propagate through Q/K/V, inducing attention. Adding explicit attention bias parameters (k′, v′) during training eliminates the massive activation phenomenon while preserving model performance.
When it emerges (ICLR 2025 spotlight)
During the Softmax normalization process, internal dependencies are created between tokens, causing them to act like a bias on the Key side. They contribute very little to the actual meaningful values.