Lithos Metal

Lithos Metal is an open-source inference engine that generates Metal megakernels for Apple silicon.

Lithos Metal is a fully open-source inference engine that generates Metal megakernels for Apple silicon to run low-latency LLM inference. It combines layer-wise mixer megakernels, GPU-resident generation control, and DSpark speculative decoding to reduce execution overhead. The engine supports models like Qwen and Llama on Apple chips such as M5, M4, and M3 series. It provides chat completions, responses, and Anthropic messages adapters with text generation, tool calls, and server-sent events streaming. Users can connect coding agents or call the API locally for optimized on-device text generation.

Lithos Metal at a glance

Best for: Software engineer
Runs on: macOS, CLI, API, open source
Pricing: Free and open source under Apache 2.0 license
Released: October 2026
Category: AI coding tool
Website: github.com/lithos-ai/lithos-metal

Key features

Generate Metal megakernels for Apple silicon
Execute DSpark speculative decoding
Provide local LLM serving and API adapters
Decompose operators into matrix tiles and attention partitions
Use layer-wise fusion for mixer megakernels
Manage GPU-resident generation control and state

Use it for

Run low-latency LLM inference on local Mac hardware
Serve models locally for coding agents and applications
Execute speculative decoding with DSpark heads
Handle chat completions and tool calls via local API

Alternatives to Lithos Metal

LiteLLM — One OpenAI-format API for 100+ LLM providers
MTPLX — It runs local models twice as fast via native MTP speculative decoding.
oMLX — It uses paged SSD KV caching to reduce agent TTFT to under 5 seconds.
llm-d — Deploys production-grade LLM serving on existing hardware clusters.
MLX LM — Runs LLMs and fine-tuning natively on Apple silicon hardware.
Rapid-MLX — Run 250 MLX-native models locally on Apple Silicon with high inference speeds.
Backburner — Extends LLM context limits and speeds up prompt reading using iPhone hardware.
AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
Mesh LLM — Run large local models across multiple machines through one OpenAI-compatible endpoint.
Enki LLM — Runs Llama-3.2 1B in pure Rust at 40 tok/s with live GPU to CPU migration.
Kapitsa — Optimized inference engine delivering high token generation speeds on Apple Silicon.
Strata Mlx — Runs 125B MoE models on Apple Silicon by streaming experts directly from disk.

All Lithos Metal alternatives