ort

ort is a Rust binding for ONNX Runtime that enables the deployment and execution of machine learning models with hardware-accelerated inference.

ort provides Rust language bindings for ONNX Runtime, a cross-platform machine learning inference engine. It allows developers to load neural network models converted into the ONNX graph format and execute them within Rust applications. The library supports various hardware accelerators including NVIDIA CUDA, TensorRT, Intel OpenVINO, Qualcomm QNN, and Huawei CANN. Developers can utilize execution providers to manage hardware acceleration or use alternative backends like tract and candle for expanded compatibility. It is utilized in production scenarios for file type detection, query language model support, and edge functions.

ort at a glance

Best for: Software engineer
Runs on: Linux, Web, open source
Pricing: Free
Released: November 2022
Category: AI coding tool
Website: ort.pyke.io

Key features

Hardware-accelerated inference via ONNX Runtime
Support for execution providers like CUDA and OpenVINO
Efficient I/O binding for input and output management
Compatibility with models from PyTorch, TensorFlow, Keras, and others
Alternative pure-Rust backends via tract and candle
Web deployment support via ort-web

Use it for

Running local ML models for biodiversity conservation
Implementing ONNX inference in WebAssembly runtimes
Executing ML models within database query languages
Powering AI edge functions
Implementing file type detection libraries

Alternatives to ort

ncnn — No runtime dependencies and supports CPU and Vulkan GPU backends.
TensorRT — Speeds up inference by 36X compared to CPU-only platforms.
MNN — It delivers high-performance inference and training with a lightweight footprint.
Resin — Builds high-performance inference engines for complex models like DeepSeek and GLM.
oMLX — It uses paged SSD KV caching to reduce agent TTFT to under 5 seconds.
Shimmy — Runs local AI models with zero Python or C++ dependencies.
Enki LLM — Runs Llama-3.2 1B in pure Rust at 40 tok/s with live GPU to CPU migration.
iwant — Spins up cloud GPUs and open models instantly via a single command.
fal — Access 1,000+ generative models and high-performance serverless GPUs.
Foundry Local — Runs AI models locally on device with complete privacy and no Azure subscription.
AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
Neptune — It combines a fast native terminal engine with built-in management for AI coding agents.

All ort alternatives