Seonglae ChoEfficient management of attention key and value memory with PagedAttention
Vllm Usages
Log probs
Streaming Requests & Realtime API in vLLM
Large language model inference has traditionally operated on a simple premise: the user submits a complete prompt (request), the model processes it, and returns
https://vllm.ai/blog/streaming-realtime
vLLM v0.6.0: 2.7x Throughput Improvement and 5x Latency Reduction
TL;DR: vLLM achieves 2.7x higher throughput and 5x faster TPOT (time per output token) on Llama 8B model, and 1.8x higher throughput and 2x less TPOT on Llama 70B model.
https://blog.vllm.ai/2024/09/05/perf-update.html
Supported Models — vLLM
https://vllm.readthedocs.io/en/latest/models/supported_models.html