llama-swap

Llama-swap runs multiple generative AI models locally and hot-swaps between them on demand using OpenAI and Anthropic compatible servers.

Llama-swap is a tool designed to run multiple generative AI models on a local machine and hot-swap between them on demand. Built in Go, it features zero dependencies and sets up using a single binary and configuration file. It works with any OpenAI and Anthropic API compatible server, supporting various inference backends like llama.cpp, vllm, stable-diffusion.cpp, and ComfyUI. The software includes a real-time web interface with a playground, token metrics, request inspection, and live log streaming. Users can deploy it via Docker, Homebrew, MacPorts, WinGet, pre-built binaries, or source code.

llama-swap at a glance

Best for: Software engineer
Runs on: Web, macOS, Windows, Linux, API, CLI, self-hosted, open source
Pricing: Free
Released: October 2024
Category: AI coding tool
Website: github.com/mostlygeek/llama-swap

Key features

On-demand model switching for local AI servers
OpenAI and Anthropic API supported endpoints
Real-time web interface with a playground
Automatic unloading of models after timeout
Docker and Podman support
API key support for access restriction
System and GPU metrics for Prometheus

Use it for

Running multiple local generative AI models on demand
Testing models using a real-time web playground
Monitoring system and GPU metrics
Managing local AI workflows with custom swap logic

Companions

Tools that articles about llama-swap recommend using with it, each with its job.

Open WebUI — Run AI on your own terms with complete data privacy and self-hosting control.
AnythingLLM — Runs fully on-device with no accounts, API keys, or token limits required.
LangChain — Framework for custom agents with memory and tools on local models.
Chroma — Local vector database for embeddings and semantic search.
Cline — Open source, any model, with approval on every step
OpenHands — Runs autonomous agents locally, in the cloud, or inside your VPC with complete control.

Alternatives to llama-swap

Llama Windows — It runs open models offline with zero configuration.
Distributed Llama — Splits model memory and computation across multiple connected home devices.
LlamaFactory — Provides a code-free CLI and Web UI for training 100+ different LLMs.
Ollama — Run open models locally or in the cloud with private prompts
Enki LLM — Runs Llama-3.2 1B in pure Rust at 40 tok/s with live GPU to CPU migration.
LLamaSharp — Runs large language models efficiently on local hardware using C#.
AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
Project Maya — Runs a 321B-parameter AI model locally across VRAM, RAM, and NVMe SSD.
Soup — Fine-tune and align models like Llama-3.1-8B on a 4 GB laptop GPU.
MTPLX — It runs local models twice as fast via native MTP speculative decoding.
LlamaCoder — Generates functional applications directly from text descriptions.
Fireworks AI — Run optimized open models at lower cost and higher throughput than closed alternatives.

All llama-swap alternatives