Grouped-query Attention

`GQA`

If these models used 4x GQA, the size and required bandwidth for KV-cache would have been 4x smaller

MHA와 MQA를 포함한 일반화라고 볼수도

GQA는 사후에 적용이 가능

KV Cache & GQA

It would be interesting to see the technical report as it may contain relevant ablation studies, but purely from the cost/performance point of view, GQA needs to be evaluated for every transformer-based LLM as the benefits are too significant to ignore.

When you ask ChatGPT to help with a task, your request is evaluated concurrently with many other requests on the same GPU, and the bandwidth is utilized more efficiently

LLM inference speed of light

In the process of working on calm, a minimal from-scratch fast CUDA implementation of transformer-based language model inference, a critical consideration was establishing the speed of light for the inference process, and measuring the progress relative to that speed of light. In this post we’ll cover this theoretical limit and its implications.

https://zeux.io/2024/03/15/llm-inference-sol

arxiv.org

https://arxiv.org/pdf/2305.13245.pdf

라마2에 적용된 추론 속도 향상 기술인 GQA(Grouped Query Attention)에 대해

데보션 (DEVOCEAN) 기술 블로그 , 개발자 커뮤니티이자 내/외부 소통과 성장 플랫폼

https://devocean.sk.com/blog/techBoardDetail.do?ID=165192&boardType=techBlog

Grouped-query Attention

GQA

KV Cache & GQA

Recommendations

`GQA`