Attention Mechanism Optimization

Creator

Creator

Seonglae Cho

Created

Created

2023 Oct 6 8:0

Editor

Editor

Seonglae Cho

Edited

Edited

2024 Oct 10 11:7

Refs

Refs

Model Quantization

Relative Positional Encoding

https://arxiv.org/pdf/1706.03762.pdf

notion image

Attention Mechanism Optimizations

Sparse Attention

Flash Attention

Dilated Attention

PagedAttention

Group Query Attentiion

Multi Query Attention

Clustered attention

Layer Selective Rank Reduction

Paged Attention

Chunk Attention

Memory-efficient Attention

Gated Attention

Selective Attention

Multi-head Attention Optimization

Multi-head Attention

Grouped-query Attention

Multi-query Attention

Sigmoid Attention, replacing the traditional softmax with a sigmoid and a constant bias

Theory, Analysis, and Best Practices for Sigmoid Self-Attention

Attention is a key part of the transformer architecture. It is a sequence-to-sequence mapping that transforms each sequence element into a weighted sum of values. The weights are typically...

Theory, Analysis, and Best Practices for Sigmoid Self-Attention

https://arxiv.org/abs/2409.04431

Theory, Analysis, and Best Practices for Sigmoid Self-Attention

How to make LLMs go fast

Blog about linguistics, programming, and my projects

How to make LLMs go fast

https://vgel.me/posts/faster-inference/

A guide to LLM inference and performance

To attain the full power of a GPU during LLM inference, you have to know if the inference is compute bound or memory bound. Learn how to better utilize GPU resources.

A guide to LLM inference and performance

https://www.baseten.co/blog/llm-transformer-inference-guide

A guide to LLM inference and performance

Optimization

Hugging Face Reads, Feb. 2021 - Long-range Transformers

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Hugging Face Reads, Feb. 2021 - Long-range Transformers

https://huggingface.co/blog/long-range-transformers

Hugging Face Reads, Feb. 2021 - Long-range Transformers

Recommendations

/////////