NVIDIA Model Optimizer

NVIDIA Model Optimizer is a deep learning library that compresses models to accelerate inference.

NVIDIA Model Optimizer is a library comprising state-of-the-art model optimization techniques including quantization, pruning, neural architecture search, distillation, speculative decoding, and sparsity. It accepts inputs from Hugging Face, PyTorch, or ONNX models and provides Python APIs to compose optimization techniques and export checkpoints. The library integrates with NVIDIA Megatron-Bridge, Megatron-LM, and Hugging Face Accelerate for training-required inference optimizations. Exported checkpoints are ready for downstream deployment in frameworks like SGLang, TensorRT-LLM, TensorRT, or vLLM. The unified Hugging Face export API supports both transformers and diffusers models.

NVIDIA Model Optimizer at a glance

Best for: Software engineer, Researcher
Runs on: Python, Linux, Windows, API, open source
Pricing: Free
Released: April 2024
Category: AI coding tool
Website: github.com/NVIDIA/Model-Optimizer

Key features

Post-training quantization to compress model size
Quantization-aware training and distillation for accuracy recovery
Pruning to reduce model parameters and memory footprint
Distillation to train smaller models using larger models
Speculative decoding for predicting extra tokens
Sparsity for storing non-zero parameter values
Python APIs for composing optimization techniques

Use it for

Compress deep learning models for downstream deployment
Accelerate generative AI inference performance
Quantize models for deployment in TensorRT-LLM and vLLM
Prune and distill large language models

Alternatives to NVIDIA Model Optimizer

TensorRT — Speeds up inference by 36X compared to CPU-only platforms.
AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
fal — Access 1,000+ generative models and high-performance serverless GPUs.
LMDeploy — Accelerates inference with high-performance engines and quantization tools.
Neural Compressor — Supports advanced quantization and sparsity for large language models.
XTuner — High-efficiency training for 200B+ scale MoE models without expert parallelism.
torchao — It speeds up training and inference while reducing memory usage with native PyTorch.
Soup — Fine-tune and align models like Llama-3.1-8B on a 4 GB laptop GPU.
ncnn — No runtime dependencies and supports CPU and Vulkan GPU backends.
Hugging Face Transformers — It provides a unified API for over 1M+ pretrained model checkpoints on the Hub.
SpectraAdamw — Reduces optimizer state memory by fifty percent without losing convergence speed.
Liger Kernel — Increases LLM training throughput by 20% and reduces memory usage by 60%.

All NVIDIA Model Optimizer alternatives