AirLLM

AirLLM is a Python library for running large language models on limited hardware by streaming model layers from disk to the GPU.

AirLLM reduces inference memory requirements by decomposing large language models and loading layers individually rather than keeping the entire model in VRAM. This process enables the execution of massive models, such as Llama 3 or 2.8T parameter models, on consumer hardware with as little as 4GB of GPU memory. The library supports automatic detection of model types through the AutoModel interface, allowing integration with major families including Llama, Qwen, DeepSeek, and Mistral. Beyond inference, it provides tools for training huge models by streaming frozen weights while keeping only adapters resident in memory. It also includes features for block-wise quantization to improve inference speed.

AirLLM at a glance

Best for: Software engineer, Researcher
Runs on: Linux, macOS
Pricing: Free
Released: June 2023
Category: AI coding tool
Website: github.com/lyogavin/airllm

Key features

Layer-wise model streaming from disk to GPU
AutoModel interface for major LLM families
Fine-tuning with streaming frozen base weights
4-bit and 8-bit block-wise quantization
Prefetching to overlap model loading and compute
CPU inference support
Automatic model type detection

Use it for

Executing 70B+ parameter models on 4GB VRAM GPUs
Fine-tuning large models on consumer graphics cards
Deploying open-source models without high-end VRAM requirements

Alternatives to AirLLM

vLLM — Maximizes throughput with PagedAttention and continuous batching.
LMDeploy — Accelerates inference with high-performance engines and quantization tools.
llm-d — Deploys production-grade LLM serving on existing hardware clusters.
Soup — Fine-tune and align models like Llama-3.1-8B on a 4 GB laptop GPU.
LightLLM — It delivers high-speed inference and serving performance for large language models.
fal — Access 1,000+ generative models and high-performance serverless GPUs.
WebLLM — Runs high-performance language models directly inside web browsers.
whichllm — Ranks models using real benchmarks and hardware fit instead of parameter count.
MLX LM — Runs LLMs and fine-tuning natively on Apple silicon hardware.
llmrun — Check which open models run on your specific GPU or Mac instantly.
Strata — Runs large AI models locally on consumer gaming PCs without server reliance.
RunThisLLM — Provides exact hardware and VRAM requirements for over 295 local AI models.

All AirLLM alternatives