open source · alpha, but real

Selfhost LLM stacks
Run the whole fleet from your terminal.

Models, vector DBs, embeddings, caching, a retrieval agent, and evals, all on your own hardware, with no third-party API keys and no data leaving your machine. It feels like docker compose up, not a CUDA scavenger hunt: a stable OpenAI-compatible API + a web UI per instance, managed as a fleet.

$ curl -fsSL llmaker.sh/install | sh
Get started →
🦙
llmaker
● fleet control
brave-llama
llama3:8b · :11500
code-wizard
qwen2.5-coder:7b
mistral-sage
mistral:7b · :11501
tiny-fox
pulling llama3.2:1b…
rag-vectors
qdrant · :6333
rag-embed
nomic-embed-text
vision-owl
llava:7b
repo-index
qdrant · :6334
tei-embed
embeddings · tei
chat-ui
open-webui · :3000
sage-mem
redis · :6379
trace
langfuse · :3001
img-index
chroma · :8001
web-search
searxng · :8888
voice-in
whisper · :9000
rag-agent
langgraph · :8000
flow
n8n · :5678
zsh · llmaker
➜ Starting brave-llama (Ollama · llama3:8b)
✓ Image ready
✓ Creating & starting brave-llama
model4.7 GB / 4.7 GB
✓ Pulled llama3:8b
╭ ✓ Instance ready ─────────────────╮
│ endpoint http://127.0.0.1:11500/v1 │
│ web UI http://127.0.0.1:11500 │
╰──────────────────────────────────────╯

One command. A real OpenAI-compatible endpoint, a browser UI, and live status, in under a minute. Then llmaker top shows load across every instance.

Why llmaker

You just wanted to run Llama 3 locally. Simple, right?

Forty browser tabs later, you're deep in a 2019 NVIDIA forum thread, your docker run has fourteen -e flags, something's already wedged on port 8080, and you still don't have a UI. llmaker is the part where you stop doing that: it treats LLM servers like services you launch, observe, and throw away, behind one stable API.

Ollama CLIdocker model runLM Studio / Jan🦙 llmaker
Multi-backend behind one API❌❌❌✅
Per-instance isolation❌⚠️❌✅
API and web UI per instance❌❌⚠️✅
Fleet view + resource limits❌❌❌✅
GPU partitioning (exclusive slices)❌❌❌✅
Full app stacks (RAG · voice · SQL)❌❌❌✅
Terminal-first (live TUI)⚠️❌❌✅
Honest about the Mac-GPU problemn/a⚠️✅✅

"htop for your local LLM fleet," plus the part where you start the fleet.

Features

Everything the fleet needs, nothing it doesn't.

🔌
Backend-agnostic
Ollama today, llama.cpp next. Switching is a --backend flag; nothing in your app changes.
🏷️
Fleet = state
Tracked entirely through Docker labels. ls can’t drift out of sync, because the containers are the source of truth.
🎨
Beautiful terminal UX
Cobra + Charm (Bubble Tea, Lip Gloss, Bubbles, Huh): a wizard, live progress bars, and llmaker top.
🌐
A UI in every box
Each instance serves its own dark-mode dashboard: gauges, model management, a chat tester, snippets.
🧠
One stable API
OpenAI-compatible /v1/* with SSE streaming for chat, completions, and embeddings. Point any client at it.
🖥️
Honest about hardware
Auto-detects GPUs; warns about the Docker-on-macOS no-Metal reality before you hit mystery latency.
📜
Declarative fleets
stack.yaml + llmaker apply: compose-like, but LLM-aware, with --prune reconcile and gang GPU admission.
🪶
Slim by choice
A ~360 MB CPU-only image (vs 8.5 GB GPU) for laptops, CI, and Macs.
🧩
Whole stacks, not just models
A full service catalog: vector DBs (Qdrant, Chroma, pgvector, Weaviate), TEI embeddings, Redis memory, Whisper voice, SearXNG search, Langfuse tracing, Open WebUI, n8n, Flowise.
🎛️
GPU partitioning
Split a multi-GPU box between instances: --gpus 1 grabs a free device exclusively, --gpus 0,1 pins IDs, --gpus all shares. Stacks are admitted as a unit.
🕸️
Zero-wiring networking
Everything joins a private llmaker-net and resolves by name (qdrant:6333) with no IP bookkeeping. network: groups isolate stacks; empty networks get garbage-collected.
🤖
Built-in retrieval agent
A LangGraph service runs rewrite → retrieve → rerank → generate, with tools (calculator, knowledge base, web search, read-only SQL) and Redis-backed session memory.
📏
Evals built in
Run question sets through your RAG pipeline and grade them: LLM-as-judge groundedness, relevance, correctness, and context recall, traced to Langfuse.
🔒
Private by default
Everything binds to 127.0.0.1, CORS is off, and exposing an instance publicly without an API key earns you a warning on every boot. Your data stays home.
GPU support

Slice a multi-GPU box between instances.

Partitions are exclusive and tracked on container labels, so no two instances silently fight over the same VRAM. All-GPU grants stay shareable, but never mix with partitions.

--gpus 1 Grab any one free GPU, exclusively yours until the instance is removed.
--gpus 0,1 Pin exactly those device IDs; nothing else can claim them.
--gpus all  (or --gpu) All GPUs, shareable. The classic mode.
gpus: 1  in stack.yaml Gang admission: llmaker apply admits a stack's GPU demand as a unit: the whole stack fits, or nothing provisions.

Passthrough needs Linux + the NVIDIA Container Toolkit; the GPU image is ~8.5 GB, the CPU-only one ~360 MB. On macOS, Apple's GPU can't cross Docker's VM, so llmaker is CPU-only there, and llmaker doctor says so out loud instead of letting you find out via mystery latency.

Stack templates

One command from model to app.

llmaker stack up rag scaffolds and applies a complete self-hosted application (model, services, networking, agent), or stack init just writes the YAML for you to edit.

assistant
Private ChatGPT-style assistant over a local model.
LLM · Open WebUI
voice
Talk to a model via self-hosted Whisper.
LLM · Open WebUI · Whisper
rag
Document Q&A with grounded answers and tracing.
LLM · Qdrant · embeddings · agent · Langfuse
research
Tool-using assistant that searches the web and your docs.
LLM · SearXNG · Qdrant · embeddings · agent
code
Code assistant that ingests repos for grounded review.
code LLM · Qdrant · embeddings · agent
chatbot
Multi-turn assistant with web UI and session memory.
LLM · Redis · agent
faq
Knowledge-base assistant for short grounded answers.
LLM · Qdrant · embeddings · agent
recommend
Semantic recommendations: "more like this".
Qdrant · embeddings · agent
sql
Ask your database questions in English, read-only.
LLM · Postgres · Qdrant · embeddings · agent
The agent under the hood
rewrite→ retrieve→ rerank→ generate

A FastAPI + LangGraph service (~510 MB image) behind every agent-backed stack. It collapses multi-turn history into a standalone query, retrieves and MMR-reranks from your vector store, and answers with sources. One HTTP surface for the rest: /api/chat · /api/agent · /api/ingest · /api/summarize · /api/extract · /api/transcribe · /api/recommend · /api/eval.

Commands

A verb for every move.

llmaker up [preset] Create + start an instance: a preset, explicit flags (--gpus, --memory), or an interactive wizard.
llmaker ls Styled fleet table; supports --json and --quiet.
llmaker top Live, animated dashboard across the fleet.
llmaker status <name> Detailed status: gauges, loaded / installed models.
llmaker pull <model> --on <name> Download a model with a live progress bar; --default sets it as the instance default.
llmaker chat [name] Interactive or one-shot chat via --message or piped stdin.
llmaker open <name> Open the instance’s web UI; --print just emits the URL.
llmaker logs <name> -f Stream container logs.
llmaker stop / start / restart / rm Lifecycle; rm --force skips confirmation.
llmaker apply -f stack.yaml Reconcile a declarative stack; --prune removes anything undeclared.
llmaker stack up <template> Scaffold + apply a full app stack: assistant, voice, rag, research, code, chatbot, faq, recommend, sql. stack init just writes the YAML.
llmaker service add <type> Provision a catalog service (qdrant, redis, whisper…). Browse with service catalog; manage with ls / rm / stop / start / restart.
llmaker doctor Environment check (Docker, GPU, the macOS caveat).
Architecture

A Go CLI, a Python facade, one HTTP contract.

Each half uses the best ecosystem for its job, decoupled by the contract. No local state file: every instance is a labeled container, so the fleet view is always reality.

llmaker CLI · Go, single static binary
Cobra + Charm TUI · Docker Go SDK · fleet via labels
create / start / stop↓
HTTP: status, pull, chat↓
CONTAINER INSTANCE
Backend engine
Ollama (llama.cpp next)
loopback only
⇄
Facade · Python / FastAPI
/v1/* · /api/status · models
web UI · /ws/status
model volume · :8080 → host :PORT · joins llmaker-net
Ollama ✅ default
Easiest path; rich model library; simple pull/run.
llama.cpp 🛣️ planned
Max control: GGUF, quantization, perf flags. Same contract, new adapter.
Metal on macOS 🛣️ planned
Apple-native inference without the Docker VM penalty.

Spin up your first instance.

Requires Docker (Go 1.25+ only if building from source). GPU acceleration needs Linux + the NVIDIA Container Toolkit; run llmaker doctor afterward to check your environment.

# prebuilt binary (Linux / macOS)
curl -fsSL https://raw.githubusercontent.com/raiyanyahya/llmaker/master/scripts/install.sh | sh
# with the Go toolchain
go install github.com/raiyanyahya/llmaker/cmd/llmaker@latest
Read the docs on GitHub ↗