llmman

A command-line tool for managing and serving LLM models using OCI registries. Models are packaged as standard OCI artifacts and stored in any compatible registry (Docker Hub, GHCR, quay, self-hosted, etc.). llmman serve exposes Ollama-, OpenAI-, and Anthropic-compatible HTTP APIs.

Commands

Command	Description
serve	Start an inference server (Ollama / OpenAI / Anthropic APIs)
launch	Launch an integration (Claude Code, OpenCode, …)
run	Run a model interactively or with a one-shot prompt
pull	Pull a model from a registry or HuggingFace
list	List locally stored models
ps	List models currently loaded
stop	Stop (unload) a running model
build	Package model files into a local OCI image
push	Push a local image to a registry
transfer	Transfer an image directly from one location to another (e.g. HuggingFace to an OCI registry)
cp	Copy a local image to a new reference
rm	Remove a local image
show	Show a local model's architecture, parameters, license, and template
login	Log in to a container registry
logout	Log out from a container registry

Install

Linux, macOS:

curl -fsSL https://raw.githubusercontent.com/llmmanorg/llmman/main/install.sh | sh

Windows (PowerShell):

irm https://raw.githubusercontent.com/llmmanorg/llmman/main/install.ps1 | iex

Quick start

Pull a model

llmman pull gemma4

Transfer a model between locations

Transfer an image directly from a source to a destination without storing it locally first, e.g. HuggingFace straight to an OCI registry:

llmman transfer hf.co/unsloth/Qwen3.5-0.8B-GGUF docker.io/owner/model:latest

Any source llmman pull understands (an OCI registry, hf://, ms://, ...) can be paired with any OCI registry destination.

Serve

Start the inference server. GGUF models are served by llama-server from llama.cpp, used from PATH if it's already there; otherwise llmman downloads and caches a prebuilt release matching your OS/arch/GPU automatically (see --llama-cpp-version to pin a specific release). Safetensors models are served by vllm (plain vllm is CPU-only on macOS, unless you separately install vllm-metal for Metal GPU support), or, on Apple Silicon macOS, by mlx-lm's mlx_lm.server instead when it's on PATH: Metal-accelerated, with no vLLM dependency at all, and it supports more model families than vllm-metal does.

llmman serve

The server listens on 127.0.0.1:17434 by default, overridable via LLMMAN_HOST, and exposes:

API	Endpoints
Ollama	/api/generate, /api/chat, /api/tags, /api/show, /api/pull, /api/ps, /api/delete
OpenAI	/v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models, /v1/responses, /v1/responses/input_tokens
Anthropic	/v1/messages

/v1/responses implements the OpenAI Responses API (the dialect OpenAI Codex requires), including streaming SSE and function-tool-call re-mapping. This is a plain pass-through to llama-server's own native /v1/responses support, so a recent enough llama-server build is required for it to work.

Use it as an Ollama-compatible server:

OLLAMA_HOST=127.0.0.1:17434 ollama run unsloth/Qwen3.5-0.8B-GGUF

Or with any Ollama, Anthropic or OpenAI-compatible client.

Models are loaded on demand. Each model gets its own llama-server subprocess on a random loopback port; subsequent requests reuse the running process.

/api/chat also supports Ollama's tools (function calling, streamed back as message.tool_calls), images (vision, base64, same as Ollama's own wire format), and format ("json" or a JSON Schema object, for constrained structured output).

An idle, unused model is automatically unloaded after keep_alive (default 5 minutes, matching Ollama; set per-request, or daemon-wide via LLMMAN_KEEP_ALIVE), and llmman ps//api/ps reports each model's expires_at.

Daemon-wide settings, set before llmman serve starts. llmman is a very different program underneath (no per-GPU memory estimator, no embedded inference engine, no cloud/desktop-app features), so an equivalent setting may not behave identically.

Variable	Effect
LLMMAN_DEBUG	Enables verbose diagnostic logging (a spawned backend's full command line, per-GPU probe detail, etc). Accepts 1/true/yes/on, or any other non-zero integer.
LLMMAN_HOST	[host][:port] llmman serve binds to. Every llmman client in the same environment connects to it too, rewriting a wildcard host to loopback first. Defaults to 127.0.0.1:17434.
LLMMAN_CONTEXT_LENGTH	Context size (--ctx-size) for every model this daemon loads. Defaults to a VRAM-tiered value when unset.
LLMMAN_KEEP_ALIVE	The daemon-wide default keep_alive (how long an idle, unused model stays loaded before being unloaded). Defaults to 5 minutes. Overridden per-request by /api/chat//api/generate's own keep_alive field.
LLMMAN_MAX_LOADED_MODELS	Caps how many models this daemon keeps loaded at once, as one flat daemon-wide total (llmman has no per-model memory estimate to size an automatic per-GPU figure against). Once at the cap, the least-recently-used idle model is evicted to make room; if every loaded model is busy, the request gets a 503 instead. Defaults to 0 (unbounded, today's behavior, unchanged).
LLMMAN_MAX_QUEUE	Caps how many requests llmman serve admits into scheduling at once; anything beyond that gets an immediate 503 (server busy, please try again. maximum pending requests exceeded, two spaces included). Defaults to 512.
LLMMAN_MAX_TRANSFER_STREAMS	Maximum number of a HuggingFace safetensors repo's files downloaded concurrently during pull. Has no effect on GGUF transfers, and is not read by transfer's own docker-feature registry-push path, which streams files sequentially. Defaults to 4.
LLMMAN_MODELS	Local store directory, overriding the default below. pull/push/run/etc. go through the daemon and always use whichever store it was started with.
LLMMAN_NUM_PARALLEL	Number of parallel request slots (--parallel) for GGUF models (llama-server only; no vllm/mlx equivalent). --ctx-size is scaled up by this value first, so each slot still gets the full configured/default context rather than an even split of it; ignored (with a warning) for a load with no explicit context size to scale. Unset leaves llama-server's own default of 1 untouched.
LLMMAN_ORIGINS	A comma-separated list of extra allowed CORS origins for the HTTP API. A trailing :* on an entry matches any port on that scheme+host. Always includes every scheme/port on localhost/127.0.0.1/0.0.0.0/[::1] regardless of this variable.
LLMMAN_SCHED_SPREAD	Truthy forwards --split-mode layer (spread a model across every GPU, already llama-server's own default); falsey forwards --split-mode none (restrict to one GPU).
LLMMAN_FLASH_ATTENTION	Flash Attention mode (--flash-attn): on, off, or auto (llama-server's own default). Also accepts 1/0/true/false.
LLMMAN_KV_CACHE_TYPE	KV-cache quantization (--cache-type-k/--cache-type-v), e.g. f16 (default), q8_0, q4_0. Trades output quality for memory at long context lengths.
LLMMAN_LLM_LIBRARY	Forces which GPU backend llmman serve/run picks (cpu, cuda/cuda12, cuda13, rocm, vulkan, or macOS-only metal), bypassing autodetection. Has no effect when a llama-server binary is already on PATH (its own backend is fixed), or on macOS's local-binary download (one asset per architecture, no separate choice to make).
LLMMAN_GPU_OVERHEAD	Bytes of VRAM to hold back from the VRAM-tiered LLMMAN_CONTEXT_LENGTH default, leaving headroom for whatever else shares the device. Applied as one combined-total subtraction rather than per-GPU (llmman only ever probes one combined VRAM total).
LLMMAN_IGPU_ENABLE	Counts integrated GPUs (Vulkan only) when probing for an accelerator. Defaults to disabled, since an integrated GPU is usually a worse choice than the discrete/CPU fallback it would otherwise be skipped in favor of.
LLMMAN_LOAD_TIMEOUT	How long to allow a model load to stall before giving up. Zero or negative means wait forever. Defaults to 10 minutes (vllm can take several minutes to load a large safetensors model).
LLMMAN_TMPDIR	Staging directory for llama-server release downloads, overriding the default tmp subdirectory of the install root.
LLMMAN_NOPRUNE	When set (to anything other than 0/false/no/off), skips the garbage-collection sweep that llmman rm and llmman serve startup otherwise run to delete blobs and extracted-cache entries no longer referenced by any local model. Note this is broader than skipping the daemon-startup catch-all: it also stops llmman rm itself from ever freeing disk space, so a removed model's (possibly multi-GB) weights stay on disk until a later sweep runs without this set. Useful for a shared/read-mostly store, or scripts that rm in a loop and prune once at the end.
LLAMA_ARG_FIT / LLAMA_ARG_FIT_TARGET	llama.cpp's own env-configurable --fit/--fit-target options. Not something llmman parses itself, just forwarded through to every llama-server (local or --ociman container) it spawns, same as CUDA_VISIBLE_DEVICES/etc. below.

GPU device-selection variables llmman serve forwards to every llama-server it spawns (local or --ociman container): CUDA_VISIBLE_DEVICES, HIP_VISIBLE_DEVICES, ROCR_VISIBLE_DEVICES, GGML_VK_VISIBLE_DEVICES, GPU_DEVICE_ORDINAL, HSA_OVERRIDE_GFX_VERSION.

Launch an integration

Point an integration at a model in one step. llmman launch starts serve in the background if it isn't already running (preloading the requested model), then sets the right environment variables and execs the integration:

llmman launch claude --model gemma4

Run llmman launch with no arguments to list the supported integrations (Claude Code, OpenCode) and whether each is installed. Any extra arguments after -- are forwarded to the integration's own CLI.

Short names work wherever a model reference is accepted.

Hosted providers

--provider points the same integrations at a model llmman doesn't serve itself:

export OPENROUTER_API_KEY=...
llmman launch opencode --provider openrouter --model qwen/qwen3-coder

The provider list is fetched at runtime from models.dev — the same catalog opencode resolves its own providers from — so a newly added provider works without an llmman release. It's cached for 24 hours, and a stale copy is used if the fetch fails, so being offline means an out-of-date list rather than a broken command. llmman launch --list-providers prints the providers llmman can route to, with each one's API-key variable and whether it's set.

Requests still go through llmman serve; --provider changes where the daemon forwards them, not who the integration talks to. So one endpoint and one place integrations are configured, whether a model is local or hosted, and both usable from the same session.

The API key is read from the variable models.dev names for that provider and travels per request, never to disk. hermes is the exception: llmman configures it through a file on disk, so it can't carry a key and llmman serve needs the variable in its own environment instead. That fallback is only used for a daemon bound to loopback, and never for a browser request from another site — it bounds the blast radius rather than authenticating anyone, so on a shared machine prefer an integration that sends its own key. cline, kimi, copilot, gemini and openclaw can't be used with --provider at all — the first two pick their own model rather than taking llmman's, copilot has no way to send a key, gemini feeds its key to a native Google client llmman can't confirm it has redirected, and openclaw only takes a model during first-run onboarding.

Being OpenAI-compatible doesn't mean implementing every OpenAI route. codex uses /v1/responses, which openai, groq and openrouter answer but anthropic and mistral don't; models.dev carries no capability data to filter on, so llmman turns that provider's bare 404 into an explanation naming what's missing rather than guessing up front.

--provider needs a local llmman serve. The daemon talks plain HTTP and has no authentication, so llmman will not hand an integration a real key to send to a remote LLMMAN_HOST, and a daemon bound to anything but loopback will not spend its own environment's key on behalf of a caller that didn't present one.

Providers llmman cannot reach with a single bearer token over an OpenAI-compatible https endpoint are deliberately absent rather than half-supported: Amazon Bedrock (SigV4 request signing), Google Vertex (GCP service-account credentials), Azure, and the others whose endpoint is per-account or whose wire format isn't OpenAI's.

Use with vLLM directly

llmman serve already spawns vllm itself as a backend for safetensors models. The vllm-llmman plugin is the inverse: install it alongside vllm and vllm serve oci://<reference> pulls a CNCF ModelPack image directly, instead of a HuggingFace repo.

MLX (Apple Silicon)

On Apple Silicon macOS, llmman serve uses mlx_lm.server instead of vllm for safetensors models whenever it's on PATH, Metal-accelerated, with no vLLM dependency at all (unlike getting the same acceleration out of vllm serve itself via vllm-metal). Falls back to vllm otherwise. Doesn't support /v1/embeddings.

Store location

Default locations:

OS	Path
Linux, macOS	~/.local/share/llmman/store
Windows	%LOCALAPPDATA%\llmman\store

Set LLMMAN_MODELS to change this (matching Ollama's OLLAMA_MODELS). Commands that read or write the local store directly (list, rm, build, serve) all honor it. Commands that go through the background daemon instead (pull, push, run, launch, ps) always use whichever store the daemon was started with; set LLMMAN_MODELS before llmman serve to change it for all of them. transfer, login, and logout never touch a local store at all.

The store uses OCI Image Layout, readable by docker and podman.

Transport backends

The registry transport is a compiled-in Go shim. Two backends are available via Cargo feature flags.

Docker (default)

Uses github.com/containerd/containerd, the same OCI resolver used by Docker.

cargo build --release

Podman

Uses github.com/podman-container-tools/container-libs, the same library Podman uses internally.

cargo build --release --no-default-features --features podman