Seonglae ChoA bigger model that actually cuts long-context inference cost significantly. The key is KV cache reduction: 3,514 bytes/token in V4-Flash → 890 bytes/token, roughly a 4x drop.
- Causal Encoder-Decoder: only about half the model runs during prefill, reducing input processing cost.
- SWA + Bounded Replay: local KV isn't stored; only the recent portion is recomputed when needed.
- CSA2: multiple layers share long-term KV and retrieval results.
- Hierarchical Sparse Indexer: instead of searching all 1M tokens at every layer, candidates are narrowed down progressively.
- Remaining KV is stored in FP4.
As a result, at 1M context KV is roughly 890MB vs 3.5GB before — about 1/4 the HBM and 1/8 the SSD cache. This is especially advantageous for long-running agents.