torchao

torchao is a PyTorch native library for model optimization, providing custom data types, quantization, and sparsification.

Torchao is a library for custom data types and model optimizations. It enables developers to quantize and sparsify weights, gradients, optimizers, and activations for both inference and training using native PyTorch. Users can pre-train large language models faster with float8 training, recover accuracy degradation using quantization-aware training, and run models with significantly less memory. The library supports various workflows and includes options for accelerated kernels through runtime dependencies. It serves developers looking to optimize models directly within the PyTorch ecosystem.

torchao at a glance

Best for: Software engineer
Runs on: open source, CLI
Pricing: Free
Released: November 2023
Category: AI coding tool
Website: pytorch.org

Key features

Float8 training support
Quantization-aware training
Weight quantization
Model sparsification
Activations optimization
Native PyTorch integration

Use it for

Pre-training large language models faster
Quantizing model weights for inference
Reducing memory usage during model execution
Recovering quantized accuracy degradation

Alternatives to torchao

AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
Floret Torch — Trains word vectors up to twenty times faster than CPU floret.
LMDeploy — Accelerates inference with high-performance engines and quantization tools.
Soup — Fine-tune and align models like Llama-3.1-8B on a 4 GB laptop GPU.
Strata — Runs large AI models locally on consumer gaming PCs without server reliance.
TensorZero — Combines an LLM gateway, observability, and optimization in one tool.
fal — Access 1,000+ generative models and high-performance serverless GPUs.
MNN — It delivers high-performance inference and training with a lightweight footprint.
ncnn — No runtime dependencies and supports CPU and Vulkan GPU backends.
llm-d — Deploys production-grade LLM serving on existing hardware clusters.
Gradio — Builds web interfaces for machine learning models using only Python code.
Ultralytics YOLO — Delivers high-speed and high-accuracy results in real-time.

All torchao alternatives