LMCache

LMCache is an infrastructure software that turns KV cache into AI-native memory for LLM inference.

LMCache pioneers KV cache infrastructure for LLM inference, turning KV cache into AI-native memory that can be stored, compressed, searched, and reused across your entire cluster. It supports major inference engines, hardware platforms, and pluggable external storage backends with no architectural changes required. Users can choose deployment patterns such as in-process or multiprocess mode, where LMCache runs as a standalone server separate from the inference engine. It integrates with tools like vLLM, Google Kubernetes Engine, Redis, and CoreWeave AI Object Storage. It is used in production by infrastructure teams, cloud providers, and open-source projects worldwide to improve inference performance and reduce redundant computation.

LMCache at a glance

Best for: Software engineer
Runs on: open source, self-hosted, Linux
Pricing: Free
Released: May 2024
Category: AI coding tool
Website: lmcache.ai

Key features

Store key-value pairs
Reuse cache across workers
Search cache data
Compress KV cache
Move cache efficiently
Observe cache operations

Use it for

Manage KV cache storage and recovery
Reduce redundant computation for repeated tokens
Improve performance for long system prompts
Run long-context and high-concurrency workloads

Alternatives to LMCache

GPTCache — It decreases expenses and enhances performance by caching LLM query results.
llm-d — Deploys production-grade LLM serving on existing hardware clusters.
LMDeploy — Accelerates inference with high-performance engines and quantization tools.
MLC LLM — Enables native optimization and deployment of AI models across platforms.
MemPalace — Achieves high recall with local-first storage and zero API calls.
vLLM — Maximizes throughput with PagedAttention and continuous batching.
Sulci Oss — Caches responses semantically to skip repeat LLM calls and reduce latency.
LightLLM — It delivers high-speed inference and serving performance for large language models.
oMLX — It uses paged SSD KV caching to reduce agent TTFT to under 5 seconds.
AI Memory — Shared memory for coding agents that works across machines and team members.
Codebase Memory MCP — High-speed indexing and persistent knowledge graphs for AI coding agents.
LangChain4j — Integrates LLMs and vector stores directly into Java applications.

All LMCache alternatives