Vllm

Creator

Creator

Seonglae Cho

Created

Created

2023 Jun 22 15:37

Editor

Editor

Seonglae Cho

Edited

Edited

2024 Sep 7 21:28

Refs

Refs

vllm-project • Updated 2024 Sep 7 20:3

Efficient management of attention key and value memory with

PagedAttention

Vllm Usages

vLLM v0.6.0: 2.7x Throughput Improvement and 5x Latency Reduction

TL;DR: vLLM achieves 2.7x higher throughput and 5x faster TPOT (time per output token) on Llama 8B model, and 1.8x higher throughput and 2x less TPOT on Llama 70B model.

https://blog.vllm.ai/2024/09/05/perf-update.html

Supported Models — vLLM

Supported Models — vLLM

https://vllm.readthedocs.io/en/latest/models/supported_models.html

Recommendations

//////