SGLang


pip install --upgrade pip
pip install uv
uv pip install "sglang[all]>=0.4.4.post1" --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python

supported models

Supported Models — SGLang

python3 -m sglang.launch_server --model-path lmms-lab/llava-onevision-qwen2-7b-ov --port=30000 --chat-template=chatml-llava

https://docs.sglang.ai/references/supported_models.html

Speculative Decoding

Speculative Decoding — SGLang

SGLang now provides an EAGLE2-based speculative decoding option. Our implementation aims to maximize speed and efficiency and is considered to be among the fastest in open-source LLM engines.

https://docs.sglang.ai/backend/speculative_decoding.html

SGLang

supported models

Backlinks

Recommendations