Enki LLM

Enki LLM is a pure Rust Transformer inference engine running Llama-3.2 1B with live GPU and CPU brain migration.

Enki LLM is a pure Rust Transformer inference engine built for developers wanting a minimalist implementation of LLM execution. It runs Llama-3.2 1B at approximately 40 tokens per second on an Nvidia Tesla T4 GPU. The software features live brain migration, allowing execution and KV-cache to hot-swap between GPU VRAM and CPU RAM mid-sentence without losing context. It achieves cross-silicon portability using direct LLVM-to-SPIR-V JIT compilation via Vulkan 1.3 and Rayon for CPU fallback. The implementation includes a Safetensors parser, tokenizer, Split-K GEMV, RoPE, RMSNorm, SwiGLU, and generation sampling written entirely from scratch.

Enki LLM at a glance

Best for: Software engineer, Researcher
Runs on: Linux, Windows, macOS, CLI, open source, self-hosted
Pricing: Free
Released: October 2026
Category: AI coding tool
Website: github.com/enkiruntime/enki_llm

Key features

Pure-rust inference engine
Live GPU and CPU brain migration
Cross-silicon portability
Minimalist codebase around 2000 LOC
Direct LLVM-to-SPIR-V JIT compilation
Vulkan 1.3 hardware dispatch
Rayon CPU fallback execution

Use it for

Run Llama-3.2 1B inference in pure Rust
Migrate model execution live between GPU and CPU
Compile Rust compute passes directly to SPIR-V
Execute transformer models across discrete GPUs and CPUs

Alternatives to Enki LLM

AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
llm-d — Deploys production-grade LLM serving on existing hardware clusters.
vLLM — Maximizes throughput with PagedAttention and continuous batching.
LLamaSharp — Runs large language models efficiently on local hardware using C#.
Resin — Builds high-performance inference engines for complex models like DeepSeek and GLM.
llama-swap — It runs multiple local AI models with zero external dependencies.
WebLLM — Runs high-performance language models directly inside web browsers.
LMDeploy — Accelerates inference with high-performance engines and quantization tools.
Shimmy — Runs local AI models with zero Python or C++ dependencies.
ort — Provides an ergonomic Rust API for running high-performance ONNX models.
whichllm — Ranks models using real benchmarks and hardware fit instead of parameter count.
Distributed Llama — Splits model memory and computation across multiple connected home devices.

All Enki LLM alternatives