Gated Attention

After attention output, the gate (sigmoid) reduces unnecessary information to near 0 in the gating process, so attention doesn't need to concentrate on the same token (e.g., the first token). This works by adding nonlinearity → solving the expressiveness limitation of the existing continuous linear structure between Value (Wv) and Output (Wo), and Query-dependent sparsity → removing unnecessary information → eliminating massive activation and

Attention Sink