whereismyai
Mooncake
Mooncake is a KVCache-centric disaggregated architecture designed to scale large-scale LLM inference and training across multi-node GPU clusters.
Mooncake is an infrastructure platform that separates prefill and decode clusters for LLM serving, utilizing underutilized CPU, DRAM, and SSD resources to form a disaggregated KV cache pool. It manages distributed KV cache and model weights while providing low-latency data movement across heterogeneous networks and accelerators. The platform includes a high-performance Transfer Engine that supports RDMA, NVLink, and various accelerator runtimes like CUDA and Ascend. It is designed for engineers and organizations running large-scale distributed inference or training tasks who need to improve cache reuse and reduce serving latency. The system integrates deeply with existing inference frameworks such as vLLM, SGLang, and TensorRT-LLM.
Mooncake at a glance
Key features
- Disaggregated KV cache pooling architecture
- High-performance Transfer Engine for data movement
- Topology-aware path selection for network routing
- RDMA-based P2P weight and cache transfer
- Elastic MoE serving support
- Support for multiple transport protocols including RDMA, TCP, and NVMe-oF
- Multi-NIC bandwidth aggregation
Use it for
- Scaling distributed LLM inference across multi-node clusters
- Decoupling multimodal encoders from language models
- Optimizing multi-node online training with distributed backends
- Managing heterogeneous data transfer between rollout and training
Alternatives to Mooncake
- llm-d — Deploys production-grade LLM serving on existing hardware clusters.
- LMCache — It stores, compresses, searches, and reuses KV caches across clusters.
- LMDeploy — Accelerates inference with high-performance engines and quantization tools.
- AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
- GPUStack — Unifies AI model serving and GPU compute management across heterogeneous hardware.
- Soup — Fine-tune and align models like Llama-3.1-8B on a 4 GB laptop GPU.
- GPTCache — It decreases expenses and enhances performance by caching LLM query results.
- SGLang — It delivers high-throughput, low-latency serving for large-scale deployments.
- fal — Access 1,000+ generative models and high-performance serverless GPUs.
- TokenSpeed — Optimized performance for agentic workloads and high-throughput LLM serving.
- oMLX — It uses paged SSD KV caching to reduce agent TTFT to under 5 seconds.
- Narwhal — It automatically hot-swaps prefill and decode roles without reloading model weights.
All Mooncake alternatives