Seonglae ChoGQA
If these models used 4x GQA, the size and required bandwidth for KV-cache would have been 4x smaller. This can be applied retroactively as a generalization that includes MHA and MQA.
KV Cache & GQA
It would be interesting to see the technical report as it may contain relevant ablation studies, but purely from the cost/performance point of view, GQA needs to be evaluated for every transformer-based LLM as the benefits are too significant to ignore.
When you ask ChatGPT to help with a task, your request is evaluated concurrently with many other requests on the same GPU, and the bandwidth is utilized more efficiently
LLM inference speed of light
In the process of working on calm, a minimal from-scratch fast CUDA implementation of transformer-based language model inference, a critical consideration was establishing the speed of light for the inference process, and measuring the progress relative to that speed of light. In this post we’ll cover this theoretical limit and its implications.
https://zeux.io/2024/03/15/llm-inference-sol
arxiv.org
https://arxiv.org/pdf/2305.13245.pdf
라마2에 적용된 추론 속도 향상 기술인 GQA(Grouped Query Attention)에 대해
데보션 (DEVOCEAN) 기술 블로그 , 개발자 커뮤니티이자 내/외부 소통과 성장 플랫폼
https://devocean.sk.com/blog/techBoardDetail.do?ID=165192&boardType=techBlog
open source architecture
The Architecture Behind Open-Source LLMs
In this article, we will cover various open-source models and the engineering bets that define each one.
https://blog.bytebytego.com/p/the-architecture-behind-open-source
