Attention Mechanism Optimization

Attention Mechanism Optimizations

Multi-head Attention Optimization

Multi-head Attention

Grouped-query Attention

Multi-query Attention

LSH Attention

Sigmoid Attention, replacing the traditional softmax with a sigmoid and a constant bias

Theory, Analysis, and Best Practices for Sigmoid Self-Attention

Attention is a key part of the transformer architecture. It is a sequence-to-sequence mapping that transforms each sequence element into a weighted sum of values. The weights are typically...

https://arxiv.org/abs/2409.04431

How to make LLMs go fast

Blog about linguistics, programming, and my projects

https://vgel.me/posts/faster-inference/

A guide to LLM inference and performance

To attain the full power of a GPU during LLM inference, you have to know if the inference is compute bound or memory bound. Learn how to better utilize GPU resources.

https://www.baseten.co/blog/llm-transformer-inference-guide

A guide to LLM inference and performance

Optimization

Hugging Face Reads, Feb. 2021 - Long-range Transformers

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

https://huggingface.co/blog/long-range-transformers

Hugging Face Reads, Feb. 2021 - Long-range Transformers

Currently, most frontier open-weight LLMs share a MoE transformer + various attention optimization architecture, and the actual performance difference comes from post-training and infrastructure design

The Architecture Behind Open-Source LLMs

In this article, we will cover various open-source models and the engineering bets that define each one.

https://blog.bytebytego.com/p/the-architecture-behind-open-source

The Architecture Behind Open-Source LLMs

Attention Mechanism Optimization

Optimization

Backlinks

Recommendations