Edge0

Edge0 is an open-source streaming mixture-of-experts inference framework that runs large models on consumer hardware.

Edge0 is an open-source streaming mixture-of-experts inference framework that generalizes SSD expert offload, Recover-LoRA, and prerouter routing prediction. It runs large sparse-mixture-of-experts models on consumer hardware by bounding peak memory to the active expert set rather than the total parameter count. A trained prerouter head predicts expert routing one step ahead so expert loads overlap the forward pass. The int4 base is frozen while LoRA adapters are trained by distillation to recover quantization loss. It includes Python, macOS, iOS, Android, and Windows runtimes, along with two model tiers named edge0-35b and edge0-8b.

Edge0 at a glance

Best for: Software engineer
Runs on: Python, macOS, iOS, Android, Windows, API, CLI, open source
Pricing: Free
Released: September 2026
Category: AI coding tool
Website: github.com/edge0-ai/edge0

Key features

Stream expert weights from storage on demand
Predict expert routing one step ahead with a prerouter
Freeze int4 base and train LoRA adapters by distillation
Support Python, macOS, iOS, Android, and Windows runtimes
Auto-detect tier from checkpoint configuration
Keep adapters unmerged as read-only safetensors files

Use it for

Run large sparse-mixture-of-experts models on consumer hardware
Execute inference across macOS, iOS, Android, and Windows devices
Serve 35B mixture-of-experts models with low memory usage

Alternatives to Edge0

BigMoeOnEdge — Runs Mixture-of-Experts models larger than device RAM by streaming experts from storage.
FreeToken — Runs 290B+ MoE models on consumer gaming PCs using elastic edge resources.
AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
MNN — It delivers high-performance inference and training with a lightweight footprint.
llm-d — Deploys production-grade LLM serving on existing hardware clusters.
GoRoute — Consumes 5.1MB of RAM with sub-10ms latency for coding agents.
DeepEP Ascend — Optimizes MoE training and inference on Huawei Ascend hardware with low overhead.
fal — Access 1,000+ generative models and high-performance serverless GPUs.
Strata — Runs large AI models locally on consumer gaming PCs without server reliance.
Judge0 — Provides secure, sandboxed code execution for over 60 programming languages.
PEFT — Reduces computational requirements and storage needs for large model training.
Evo — Routes traffic using custom evaluations rather than global benchmarks.

All Edge0 alternatives