whereismyai
Shimmy
Shimmy is an OpenAI-compatible local inference server for GGUF models written in pure Rust.
Shimmy is a single-binary inference server for GGUF models that provides an OpenAI-compatible API for local AI tools. Powered by the Airframe pure-Rust WebGPU transformer engine, it runs on NVIDIA, AMD, Intel, and Apple Silicon GPUs without requiring a Python runtime or C++ toolchain. The software supports 12 model families and 26 certified model and quantization combinations, featuring F32 accumulation precision and deterministic output. It is designed for developers who want a lightweight, fast alternative to heavier local inference stacks. Configuration options include YaRN RoPE scaling for extended context and TurboShimmy INT4 KV cache compression.
Shimmy at a glance
- Best for: Software engineer
- Runs on: WebGPU, self-hosted
- Pricing: Free forever; sponsorships available from $5/month to $500/month
- Released: August 2025
- Category: AI coding tool
- Website: github.com/Michael-A-Kuykendall/shimmy
Key features
- OpenAI-compatible API endpoints
- GGUF model loading
- WebGPU compute shaders
- Deterministic output generation
- INT4 KV cache compression
- YaRN RoPE scaling
Use it for
- Running local large language models
- Connecting AI tools to local models
- Executing chat completions offline
- Testing models with deterministic output
Alternatives to Shimmy
- Reflex — It is optimized for cold-start latency and single-tenant decisions.
- Foundry Local — Runs AI models locally on device with complete privacy and no Azure subscription.
- jevgrep — Reduces coding agent costs by providing targeted, relevant repository context.
- Resin — Builds high-performance inference engines for complex models like DeepSeek and GLM.
- AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
- Strata — Runs large AI models locally on consumer gaming PCs without server reliance.
- fal — Access 1,000+ generative models and high-performance serverless GPUs.
- Flare Local Inference — Optimizes memory and bandwidth for workstations and compact systems.
- Enki LLM — Runs Llama-3.2 1B in pure Rust at 40 tok/s with live GPU to CPU migration.
- ort — Provides an ergonomic Rust API for running high-performance ONNX models.
- GPUStack — Unifies AI model serving and GPU compute management across heterogeneous hardware.
- Dlgo — Run LLM inference quickly in pure Go without Python or external runtimes.
All Shimmy alternatives