llm-d

llm-d is an open-source, Kubernetes-native stack for distributed LLM inference that runs models on various accelerators.

llm-d is an open-source, Kubernetes-native stack designed for distributed LLM inference. It integrates with engines like vLLM and SGLang to convert single-node engines into production-grade serving environments across existing clusters. The platform supports diverse hardware, including GPUs, Google TPUs, Intel XPUs, CPUs, and emerging NPUs. It features pre-configured deployment recipes called Well-Lit Paths for managing production patterns. The tool aims to provide consistent performance across heterogeneous hardware, enabling teams to scale with predictable behavior. It is built for infrastructure engineers and teams managing LLM workloads on Kubernetes.

llm-d at a glance

Best for: Software engineer
Runs on: Kubernetes
Pricing: Free
Released: April 2025
Category: AI coding tool
Website: llm-d.ai

Key features

Kubernetes-native deployment architecture
Support for vLLM and SGLang engines
Deployment recipes for production patterns
Tiered prefix-cache configuration
Prefill and decode disaggregation
Wide expert parallelism for MoE models
Flow control and fairness application
Autoscaling for inference pools

Use it for

Deploying optimized LLM inference baselines
Routing requests based on predicted latency
Scaling Mixture-of-Experts models
Managing production LLM serving on heterogeneous hardware

Alternatives to llm-d

LMDeploy — Accelerates inference with high-performance engines and quantization tools.
MLC LLM — Enables native optimization and deployment of AI models across platforms.
LightLLM — It delivers high-speed inference and serving performance for large language models.
AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
Mesh LLM — Run large local models across multiple machines through one OpenAI-compatible endpoint.
vLLM — Maximizes throughput with PagedAttention and continuous batching.
OpenLLM — It simplifies infrastructure management and optimizes inference speed and cost.
WebLLM — Runs high-performance language models directly inside web browsers.
llmware — Build private, local RAG pipelines with 300+ prepackaged models and integrated document parsing.
LMCache — It stores, compresses, searches, and reuses KV caches across clusters.
Distributed Llama — Splits model memory and computation across multiple connected home devices.
Dlgo — Run LLM inference quickly in pure Go without Python or external runtimes.

All llm-d alternatives