Resin

Resin is an experimental library and runtime for building custom LLM inference engines and optimized hardware kernels.

Resin is an experimental Python and Rust library designed for building LLM inference engines from scratch. It provides a headless engine featuring a fast batch scheduler, unified slab-based cache pool, speculative decoding, and zero-overhead overlap scheduling. Users write forward passes as model-specific kernels optimized for their target hardware. The platform includes optional HTTP serving, Prometheus metrics, and native recipes for prompt rendering and parsing. It supports parallelisms such as tensor, expert, context, and data-parallel attention, alongside prefill and decode CUDA graphs. The library is intended for developers and researchers who want to experiment with custom language model inference systems and hardware configurations.

Resin at a glance

Best for: Software engineer
Runs on: Python, Rust, CLI
Pricing: Free
Released: October 2026
Category: AI coding tool
Website: github.com/ekzhang/resin

Key features

Rust batch scheduler
Unified slab-based cache pool
Speculative decoding
Prometheus metrics export
Prefill and decode CUDA graphs
NUMA pinning on worker startup
OpenAI and Anthropic HTTP endpoints

Use it for

Building custom LLM inference engines
Optimizing model kernels for specific hardware
Benchmarking large language model performance
Serving models via HTTP endpoints

Companions

Tools that articles about Resin recommend using with it, each with its job.

DeepSeek — Built to work with DeepSeek.
Cline — Open source, any model, with approval on every step
Cursor — Autocomplete, inline edits and parallel agents in one editor
OpenRouter — Access over 500 models and 80 providers through one unified API.
Ollama — Run open models locally or in the cloud with private prompts

Alternatives to Resin

llm-d — Deploys production-grade LLM serving on existing hardware clusters.
AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
LMDeploy — Accelerates inference with high-performance engines and quantization tools.
LightLLM — It delivers high-speed inference and serving performance for large language models.
fal — Access 1,000+ generative models and high-performance serverless GPUs.
Reflex — It is optimized for cold-start latency and single-tenant decisions.
ZML — It enables Python-free inference while abstracting hardware complexities.
vLLM — Maximizes throughput with PagedAttention and continuous batching.
Infernix — Serves massive Qwen models quickly on a single consumer GPU with advanced caching.
Flare Local Inference — Optimizes memory and bandwidth for workstations and compact systems.
Shimmy — Runs local AI models with zero Python or C++ dependencies.
OpenLLM — It simplifies infrastructure management and optimizes inference speed and cost.

All Resin alternatives