whereismyai
Neural Compressor
Neural Compressor is an open-source Python library that provides model compression techniques for deep learning frameworks.
Intel Neural Compressor is an open-source Python library that supports popular model compression techniques on mainstream deep learning frameworks including PyTorch, TensorFlow, and JAX. It provides static quantization, dynamic quantization, smoothquant, weight-only quantization, quantization-aware training, and mixed precision. The library supports advanced quantization of large language models and vision-language models such as LLaMA, Qwen, DeepSeek, Flux, and FramePack. It integrates with AutoRound to support diverse low-precision data types like FP8, INT8, and INT4. It runs across a wide range of hardware including Intel Gaudi accelerators, Core Ultra processors, Xeon processors, and data center GPUs, with limited testing on AMD, ARM, and NVIDIA hardware.
Neural Compressor at a glance
Key features
- Static and dynamic quantization
- Weight-only quantization
- Quantization-aware training
- Mixed precision support
- SmoothQuant implementation
- AutoRound integration
- FP8 and INT4 data types
Use it for
- Compress large language models
- Quantize vision-language models
- Optimize deep learning inference
- Reduce model memory footprint
Alternatives to Neural Compressor
- MNN — It delivers high-performance inference and training with a lightweight footprint.
- ncnn — No runtime dependencies and supports CPU and Vulkan GPU backends.
- LMDeploy — Accelerates inference with high-performance engines and quantization tools.
- AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
- Strata — Runs large AI models locally on consumer gaming PCs without server reliance.
- JFrog Boost — Lowers token usage for AI coding agents through input filtering.
- NVIDIA Model Optimizer — Compress deep learning models to accelerate inference performance.
- TensorRT — Speeds up inference by 36X compared to CPU-only platforms.
- torchao — It speeds up training and inference while reducing memory usage with native PyTorch.
- Billion Context — Compresses conversation history to run single AI coding sessions for days.
- TensorZero — Combines an LLM gateway, observability, and optimization in one tool.
- Cube Studio — An open source, end-to-end platform for MLOps and distributed model training.
All Neural Compressor alternatives