Sparse Attention

Pattern is important

Window attention, where only the most recent KVs are cached, is a natural approach

Sparse Attentions

Native Sparse Attention

DeepSeek Sparse Attention

Sliding window attention

BigBird attention

LSG attention

Dynamic Sparse Attention

Star Attention

Hash Attention

Ball Attention

Hugging Face Reads, Feb. 2021 - Long-range Transformers

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

https://huggingface.co/blog/long-range-transformers

Hugging Face Reads, Feb. 2021 - Long-range Transformers

LLMs May Not Need Dense Self Attention

Sink Tokens and the Sparsity of Attention Scores in Transformer Models

https://medium.com/@buildingblocks/llms-may-not-need-dense-self-attention-1fa3bf47522e

Sparse attention indexer for GLM 5.2

GLM-5: from Vibe Coding to Agentic Engineering

We present GLM-5, a next-generation foundation model designed to transition the paradigm of vibe coding to agentic engineering. Building upon the agentic, reasoning, and coding (ARC) capabilities...

https://arxiv.org/abs/2602.15763

IndexCache

Based on the observation that adjacent layers in sparse-attention models select 70–100% of the same tokens, IndexCache runs its own indexer only on Full layers, while the remaining Shared layers reuse the top‑k indices from the nearest Full layer. The training-free variant uses greedy search to decide which layers keep an indexer, so no weight updates are needed. The training-aware variant trains the indexer with a multi-layer distillation loss. On a 30B model, it removes 75% of indexer computation and achieves 1.82× faster prefill and 1.48× faster decoding. Patches for SGLang and vLLM are also provided.

IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse

Long-context agentic workflows have emerged as a defining use case for large language models, making attention efficiency critical for both inference speed and serving cost. Sparse attention...

https://arxiv.org/abs/2603.12201

KV Cache Transform Coding (KVTC)

This applies principles similar to JPEG compression to LLM memory, combining a pipeline of PCA-based feature decorrelation, adaptive quantization, and entropy coding. The PCA basis matrix is computed once on a calibration dataset and can then be reused for all subsequent inference.

KV Cache Transform Coding for Compact Storage in LLM Inference

Serving large language models (LLMs) at scale necessitates efficient key-value (KV) cache management. KV caches can be reused across conversation turns via shared-prefix prompts that are common in...

https://arxiv.org/abs/2511.01815

Sparse Attention

Pattern is important

IndexCache

KV Cache Transform Coding (KVTC)

Backlinks

Recommendations