GPTCache

GPTCache: Semantic cache library for large language models that stores LLM responses to reduce API costs and latency.

GPTCache is a library dedicated to building a semantic cache for storing LLM responses. As applications grow, LLM API calls become expensive and slow, and GPTCache addresses this by caching query results to reduce requests and tokens sent to the LLM. It uses embedding algorithms to convert queries into embeddings and performs similarity searches on them using a vector store, which increases cache hit probability compared to exact match systems. The system features a modular design for custom caching and integrates with LangChain, LlamaIndex, OpenAI, and Replicate. It is designed for developers building LLM applications who want to improve scalability, lower costs, and establish an adaptable development and testing environment.

GPTCache at a glance

Best for: Software engineer
Runs on: Python, Docker, open source
Pricing: Free
Released: March 2023
Category: AI coding tool
Website: github.com/zilliztech/GPTCache

Key features

Store LLM responses in a semantic cache
Convert queries into embeddings for similarity search
Support exact and similar matching with caching
Configure temperature parameters to control cache skipping
Integrate with LangChain and LlamaIndex
Provide a modular design for custom caching implementations
Support distributed caching and horizontal scaling

Use it for

Reduce LLM API costs through response caching
Improve response times for frequent queries
Test LLM applications without connecting to live services
Scale applications to bypass rate limits

Alternatives to GPTCache

LMCache — It stores, compresses, searches, and reuses KV caches across clusters.
Sulci Oss — Caches responses semantically to skip repeat LLM calls and reduce latency.
llm-d — Deploys production-grade LLM serving on existing hardware clusters.
LangChain4j — Integrates LLMs and vector stores directly into Java applications.
LightLLM — It delivers high-speed inference and serving performance for large language models.
LMDeploy — Accelerates inference with high-performance engines and quantization tools.
LitGPT — Readable single-file LLM code with tested finetuning recipes
llmware — Build private, local RAG pipelines with 300+ prepackaged models and integrated document parsing.
Lcc — Cuts token counts significantly without losing exact session details.
gptel — Direct LLM interaction inside the Emacs text editor.
GoRoute — Consumes 5.1MB of RAM with sub-10ms latency for coding agents.
GPUStack — Unifies AI model serving and GPU compute management across heterogeneous hardware.

All GPTCache alternatives