BigMoeOnEdge

BigMoeOnEdge is an adaptive streaming optimizer for Mixture-of-Experts models that runs large language models larger than RAM on edge devices.

BigMoeOnEdge is an adaptive streaming optimizer for Mixture-of-Experts models that keeps in memory what every token needs and streams the rest from storage. It measures the machine including memory, storage speed, CPU, GPU, and NPU, and builds an execution plan for that exact device without requiring manual flags or tuning. It supports any hardware and OS, loading CPU, GPU, and NPU backends as modules. It works with any MoE GGUF model that llama.cpp loads, reading the model header before download to determine execution. Every device serves the models it runs through a local OpenAI-compatible server so that chat clients and coding agents can connect locally.

BigMoeOnEdge at a glance

Best for: Software engineer
Runs on: Android, macOS, Windows, Linux, iOS, CLI, open source
Pricing: Free
Released: October 2026
Category: AI coding tool
Website: github.com/helldez/bigmoeonedge

Key features

Adaptive streaming optimizer
Local OpenAI-compatible server
Hardware and OS detection
Support for any MoE GGUF model
CPU, GPU, and NPU backend modules
Network sharing with other devices

Use it for

Run large language models larger than device RAM
Serve local AI models to chat clients and coding agents
Stream Mixture-of-Experts models from storage
Benchmark models on diverse edge hardware

Alternatives to BigMoeOnEdge

Edge0 — Stream expert weights from storage on demand to bound peak memory usage.
Strata — Runs large AI models locally on consumer gaming PCs without server reliance.
AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
llm-d — Deploys production-grade LLM serving on existing hardware clusters.
Mesh LLM — Run large local models across multiple machines through one OpenAI-compatible endpoint.
MNN — It delivers high-performance inference and training with a lightweight footprint.
MTPLX — It runs local models twice as fast via native MTP speculative decoding.
ModLens — Converts screenshots and charts into structured JSON data for coding agents.
GoRoute — Consumes 5.1MB of RAM with sub-10ms latency for coding agents.
LMDeploy — Accelerates inference with high-performance engines and quantization tools.
Soup — Fine-tune and align models like Llama-3.1-8B on a 4 GB laptop GPU.
FreeToken — Runs 290B+ MoE models on consumer gaming PCs using elastic edge resources.

All BigMoeOnEdge alternatives