Vramkit

Vramkit provides VRAM compression for LLM inference using a lossless weight codec and drop-in PyTorch KV-cache compression.

Vramkit provides two techniques to reduce GPU memory during LLM inference: model weight compression and KV-cache compression. The weight codec is a 32-bit range coder with a 2-D adaptive context model that achieves lossless compression outperforming zstd and xz on real quantized LLM weights. The KV-cache compression applies per-channel quantization on keys and per-token quantization on values, achieving near-lossless perplexity at higher compression ratios. The module is device-agnostic, running on PyTorch for CPU, Apple MPS, NVIDIA, and AMD ROCm stacks. It is built for developers and researchers optimizing large language model inference costs and context lengths.

Vramkit at a glance

Best for: Software engineer
Runs on: Python, PyTorch, CPU, macOS
Pricing: MIT
Released: October 2026
Category: AI coding tool
Website: github.com/srkrz23/vramkit

Key features

Lossless 32-bit range coder with 2-D adaptive context model
Drop-in PyTorch KV-cache compression module
Per-channel key and per-token value quantization
Device-agnostic execution for CPU, ROCm, CUDA, and MPS
Runnner validation scripts for real-model perplexity
Bit-exact verified weight compression benchmarks

Use it for

Compressing LLM weights losslessly to save VRAM
Optimizing KV-cache memory for long-context inference
Running memory-efficient inference on AMD ROCm or Apple Silicon
Benchmarking quantization perplexity on real prose models

Alternatives to Vramkit

AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
llmrun — Check which open models run on your specific GPU or Mac instantly.
LMDeploy — Accelerates inference with high-performance engines and quantization tools.
llm-d — Deploys production-grade LLM serving on existing hardware clusters.
LMCache — It stores, compresses, searches, and reuses KV caches across clusters.
Soup — Fine-tune and align models like Llama-3.1-8B on a 4 GB laptop GPU.
oMLX — It uses paged SSD KV caching to reduce agent TTFT to under 5 seconds.
fal — Access 1,000+ generative models and high-performance serverless GPUs.
Resin — Builds high-performance inference engines for complex models like DeepSeek and GLM.
Strata — Runs large AI models locally on consumer gaming PCs without server reliance.
GPUStack — Unifies AI model serving and GPU compute management across heterogeneous hardware.
Taco — Guarantees identical compressed output across any CPU and GPU architecture.

All Vramkit alternatives