Vllm Semantic Router

System-brain routing: signal-led, entropy-aware, ruthlessly clear.

Intelligent request routing—selecting the right model for each request—has become a core systems challenge. Prior work (RouteLLM, RouterDC, AutoMix, etc.) typically focuses on model selection in isolation, and does not provide an integrated framework that also covers signal extraction, safety enforcement, multi-provider management, and plugin extensibility. vLLM Semantic Router proposes a signal-driven decision-routing framework inspired by Shannon’s information theory and Boolean algebra.

Sits between the real world and LLMs, routing each request to the most suitable model. The project aims to capture signals missing from the request/response/context and combine them to make better decisions. Critical take: the scope is extremely broad, spanning routing, caching, hallucination detection, PII protection, jailbreak defense, and self-learning. Key components include simulation via a Fleet Simulator, a model-training pipeline, and a proposal system. The approach is to manage and simulate many LLM instances as a single fleet.

The core methodology is composable signal orchestration, organized into a three-layer architecture: Signal Extraction, Decision Evaluation, and Plugin Execution. In the Signal Extraction layer, a request r is mapped into 13 signal types; each signal rule outputs a binary match indicator plus a confidence score. Heuristic signals (keywords, context length, language, authz) run in sub-millisecond time, while learned signals (embeddings, domain, complexity, etc.) are handled by a LoRA-based classifier in ~10–120 ms. The Decision Engine evaluates Boolean formulas over signal conditions, shrinking routing entropy from a uniform prior of log2(K) bits to near zero. For closed-loop adaptivity, it connects to a contextual bandit framework with an O(sqrt(T)) regret bound. The embedding-similarity signal matches using a maximum cosine-similarity threshold, and the complexity signal uses contrastive scoring to measure the gap between hard vs. easy exemplar sets. Routing policies are defined in a typed neuro-symbolic DSL and can be compiled to multiple targets (YAML, Kubernetes CRDs, Helm charts, etc.).

vLLM Semantic Router: Signal Driven Decision Routing for...

As large language models (LLMs) diversify across modalities, capabilities, and cost profiles, the problem of intelligent request routing -- selecting the right model for each query at inference...