LMDeploy

LMDeploy is an LLM toolkit that provides engines for compressing, deploying, and serving large language models and vision-language models.

LMDeploy is a toolkit designed for the compression, deployment, and serving of large language models and vision-language models. It features two distinct inference engines: TurboMind, which focuses on high-performance optimization, and a PyTorch-based engine that prioritizes developer accessibility. The toolkit supports weight-only and KV cache quantization, including 4-bit inference, to improve model performance and efficiency. Capabilities include tensor parallelism, continuous batching, and dynamic split-and-fuse techniques to boost throughput. Users can deploy models via an offline batch inference pipeline or as a serving service across multiple machines and cards. It supports a wide array of models from HuggingFace, ModelScope, and openMind Hub, including Llama, Qwen, DeepSeek, and InternLM series.

LMDeploy at a glance

Best for: Software engineer
Runs on: Windows, Linux, API, CLI, self-hosted, open source
Pricing: Free
Released: June 2023
Category: AI coding tool
Website: github.com/InternLM/lmdeploy

Key features

Weight-only and KV cache quantization
Continuous batching and blocked KV cache management
Tensor parallelism for multi-GPU deployment
Multi-model, multi-machine, and multi-card inference serving
Automatic Prefix Caching
High-performance CUDA kernels
Dynamic split-and-fuse inference

Use it for

Serving LLMs and VLMs for production applications
Optimizing model inference performance via quantization
Running offline batch inference pipelines
Distributing inference services across multi-machine clusters

Alternatives to LMDeploy

llm-d — Deploys production-grade LLM serving on existing hardware clusters.
AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
vLLM — Maximizes throughput with PagedAttention and continuous batching.
LightLLM — It delivers high-speed inference and serving performance for large language models.
MLC LLM — Enables native optimization and deployment of AI models across platforms.
FastDeploy — Provides high-performance production deployment for LLMs and VLMs across multiple hardware platforms.
WebLLM — Runs high-performance language models directly inside web browsers.
MLX LM — Runs LLMs and fine-tuning natively on Apple silicon hardware.
fal — Access 1,000+ generative models and high-performance serverless GPUs.
llmware — Build private, local RAG pipelines with 300+ prepackaged models and integrated document parsing.
Distributed Llama — Splits model memory and computation across multiple connected home devices.
LMCache — It stores, compresses, searches, and reuses KV caches across clusters.

All LMDeploy alternatives