Reflex

Reflex is an inference engine that executes GGUF models on NVIDIA GPUs using Rust and CUDA.

Reflex is a local inference engine built for cold starts and real-time decision loops. It processes one request at a time with no batching or internal network server. It supports GGUF format checkpoints across multiple architectures including Qwen3, Llama, Mistral, and DeepSeek-V2. Weights are held as f16 on the GPU by default or kept quantized with a flag. It is designed for single-shot work like tool calls and classifications on NVIDIA GPUs.

Reflex at a glance

Best for: Software engineer
Runs on: Web, Docker, AWS, Kubernetes, API, self-hosted, open source
Pricing: Free
Released: September 2026
Category: AI coding tool
Website: github.com/lateos-ai/reflex

Key features

Compiles CUDA kernels ahead of time
Supports GGUF model files
Processes single requests without batching
Provides an OpenAI-compatible HTTP sidecar
Offers C FFI and local IPC embeddings
Keeps weights as f16 on the GPU by default

Use it for

Run bursty single-shot inference on serverless GPUs
Execute real-time agent decision loops
Perform classifications and tool calls

Alternatives to Reflex

Dlgo — Run LLM inference quickly in pure Go without Python or external runtimes.
Strata — Runs large AI models locally on consumer gaming PCs without server reliance.
TensorRT — Speeds up inference by 36X compared to CPU-only platforms.
Flare Local Inference — Optimizes memory and bandwidth for workstations and compact systems.
Shimmy — Runs local AI models with zero Python or C++ dependencies.
Infernix — Serves massive Qwen models quickly on a single consumer GPU with advanced caching.
Resin — Builds high-performance inference engines for complex models like DeepSeek and GLM.
Narwhal — It automatically hot-swaps prefill and decode roles without reloading model weights.
fal — Access 1,000+ generative models and high-performance serverless GPUs.
GPUStack — Unifies AI model serving and GPU compute management across heterogeneous hardware.
audio.cpp — Runs modern audio models locally with high-performance, native C++ execution.
AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.

All Reflex alternatives