SGLang

SGLang is an Apache-2.0 serving engine held by LMSYS, a non-profit, fronting language, multimodal, embedding, reward and diffusion models behind OpenAI-compatible endpoints. Its mechanism is RadixAttention: KV cache for shared prompt prefixes lives in a radix tree, so a long system prompt, a few-shot header or a repeating agent scaffold is computed once and reused across concurrent requests until evicted. Around it sit a zero-overhead CPU scheduler, continuous batching, paged attention, prefill-decode disaggregation, speculative decoding, four axes of parallelism, structured output via compressed finite-state machines, FP4/FP8/INT4/AWQ/GPTQ quantization and multi-LoRA batching. Install with pip, uv or Docker against CUDA 12.9 or 13.0; it runs on NVIDIA, AMD MI300/MI355, Intel Xeon, Google TPU and Ascend NPU.

The claim is throughput per GPU-hour; the project states deployment on 400,000-plus GPUs and integration with major RL frameworks — self-reported, so directional rather than audited; no adopting company is named on the pages read. Cadence is fast (v0.5.20 published 2026-09-18, repo pushed 2026-09-20) and the tuning surface is deep enough that someone has to own it. Two caveats: prefix caching only pays where prompts genuinely share a prefix, so single-shot unrelated queries gain little over any other engine; and there is no hosted SGLang and no price list — support is Slack, Discord and GitHub Discussions, with consulting by email. Free software whose real cost is your GPU time, your on-call rota and the flags you learn.