Attention Sink

Vertical strong line to
System Prompt or
BOS Token (>50%) in
Attention Score

LLMs aggregate information through Pyramidal Information Funneling where attention is scattering widely in lower layers and ultimately focusing on critical tokens.

Attention Sinks Notion

Streamling LLM

REGISTERS Tokens

Attention Canceling

Attention sink

arxiv.org

https://arxiv.org/pdf/2402.09221

Efficient Streaming Language Models with Attention Sinks

Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly,...

https://arxiv.org/abs/2309.17453

🕳️ Attention Sinks in LLMs for endless fluency

A Blog post by Tom Aarsen on Hugging Face

https://huggingface.co/blog/tomaarsen/attention-sinks

Null attention

arxiv.org

https://arxiv.org/pdf/1906.04284

Massive Activation

Special token, delimiter, conjunction, preposition, first token, number token, weak semantics

arxiv.org

https://arxiv.org/pdf/2402.17762

Attention Sink

Vertical strong line to System Prompt or BOS Token (>50%) in Attention Score

Attention sink

Null attention

Massive Activation

Recommendations

Vertical strong line to
System Prompt or
BOS Token (>50%) in
Attention Score