Seonglae ChoWindow attention, where only the most recent KVs are cached, is a natural approach
Sparse Attentions
Hugging Face Reads, Feb. 2021 - Long-range Transformers
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
https://huggingface.co/blog/long-range-transformers
LLMs May Not Need Dense Self Attention
Sink Tokens and the Sparsity of Attention Scores in Transformer Models
https://medium.com/@buildingblocks/llms-may-not-need-dense-self-attention-1fa3bf47522e
Sparse attention indexer for GLM 5.2
GLM-5: from Vibe Coding to Agentic Engineering
We present GLM-5, a next-generation foundation model designed to transition the paradigm of vibe coding to agentic engineering. Building upon the agentic, reasoning, and coding (ARC) capabilities...
https://arxiv.org/abs/2602.15763
