llmhub

Unified, provider-agnostic Go client for modern Large Language Models (LLMs). llmhub wraps multiple vendors (OpenAI, Anthropic, Gemini, Ollama, OpenRouter, Z.AI, and your own) behind a single, expressive API that understands multi-modal messages, streaming, and provider registries.

Why llmhub?

One API, many vendors – swap providers without rewriting your business logic.
Multi-modal ready – mix text and images in both requests and responses.
Tool calling – declare provider-agnostic tools and handle normalized tool calls.
Streaming made simple – consume deltas through idiomatic Go channels.
Extensible registry – register first-party or external providers at runtime.
Functional options – configure models, endpoints, and credentials cleanly.

Installation

go get github.com/smhanov/llmhub

Quick Start

package main

import (
    "context"
    "fmt"

    "github.com/smhanov/llmhub"
    _ "github.com/smhanov/llmhub/providers/openai"
)

func main() {
    client, err := llmhub.New("openai", "sk-YOUR-KEY", llmhub.WithModel("gpt-4o-mini"))
    if err != nil {
        panic(err)
    }

    prompt := []*llmhub.Message{
        llmhub.NewSystemMessage(llmhub.Text("You are a witty assistant.")),
        llmhub.NewUserMessage(llmhub.Text("Explain quantum mechanics in five words.")),
    }

    resp, err := client.Generate(context.Background(), prompt)
    if err != nil {
        panic(err)
    }

    fmt.Println(resp.Text())
}

Streaming Responses

stream, err := client.Stream(ctx, prompt)
if err != nil {
    // HTTP 4xx/5xx from the provider, including 429, fail here so callers
    // can fail over before writing downstream streaming headers.
    log.Fatal(err)
}
for chunk := range stream {
    if chunk.Err != nil {
        log.Printf("stream error: %v", chunk.Err)
        break
    }
    if chunk.ReasoningDelta != "" {
        log.Printf("reasoning delta: %s", chunk.ReasoningDelta)
    }
    fmt.Print(chunk.Delta)
    if chunk.Done {
        break
    }
}

Vision & Multi-modal Inputs

prompt := []*llmhub.Message{
    llmhub.NewUserMessage(
        llmhub.Text("What is shown here?"),
        llmhub.Image("https://example.com/diagram.png"),
    ),
}
resp, _ := client.Generate(ctx, prompt)
for _, part := range resp.Content {
    if text, ok := part.(*llmhub.TextContent); ok {
        fmt.Println(text.Text)
    }
}

Reasoning / Thinking Blocks

Some models expose reasoning as separate blocks in the response payload. llmhub preserves those blocks in Response.Content as *llmhub.ReasoningContent.

resp, _ := client.Generate(ctx, prompt)

fmt.Println("final answer:", resp.Text())
fmt.Println("reasoning:", resp.ReasoningText())

for _, part := range resp.Content {
    if r, ok := part.(*llmhub.ReasoningContent); ok {
        fmt.Println("reasoning block:", r.Text)
    }
}

For streaming, reasoning is exposed separately on each chunk via StreamChunk.ReasoningDelta.

Thinking Level

Request-side reasoning depth is controlled with WithThinkingLevel (minimal, low, medium, high, xhigh). Unset means the model default. Each provider maps it to its native control; providers without one ignore it.

client, _ := llmhub.New("openai", apiKey,
    llmhub.WithModel("o3-mini"),
    llmhub.WithThinkingLevel(llmhub.ThinkingLevelMedium),
)

Level	Gemini (3.x+)	OpenAI / xAI / OpenRouter / ZAI	Anthropic (4.6+)	Ollama
unset	omit	omit	omit	omit
`minimal`	`minimal`	`reasoning_effort: minimal`	`low`	`think: false`
`low` / `medium` / `high`	1:1	1:1	1:1 (`output_config.effort`)	`think: "low"` / `"medium"` / `"high"`
`xhigh`	`high` (clamp)	`xhigh`	`xhigh`	`think: "high"` (clamp)

Gemini 2.5-series models do not accept thinkingLevel — they use the deprecated thinkingBudget control, which llmhub does not send — so setting ThinkingLevel on a 2.5 model returns upstream HTTP 400 "Thinking level is not supported for this model."; leave ThinkingLevel unset for the 2.5 series. Level support also varies within 3.x/4.x: minimal is rejected by gemini-3.8-flash and gemini-4-argon, and some models accept only a subset of levels (gemini-3-pro-preview accepts only low and high). See the Gemini thinking docs (https://ai.google.dev/gemini-api/docs/generate-content/thinking) for the per-model table.

Tool Calling

Declare tools with WithTools, read requested calls from Response.ToolCalls(), execute them in your code, then send the result back with NewToolResultMessage.

weather := llmhub.NewTool("weather", "Get current weather", map[string]interface{}{
    "type": "object",
    "properties": map[string]interface{}{
        "city": map[string]interface{}{"type": "string"},
    },
    "required": []string{"city"},
})

client, _ := llmhub.New("openai", apiKey,
    llmhub.WithModel("gpt-4o-mini"),
    llmhub.WithTools(weather),
    llmhub.WithToolChoice(llmhub.AutoToolChoice()),
)

messages := []*llmhub.Message{
    llmhub.NewUserMessage(llmhub.Text("What is the weather in Toronto?")),
}

resp, _ := client.Generate(ctx, messages)
for _, call := range resp.ToolCalls() {
    result := runTool(call.Name, call.Arguments)
    messages = append(messages,
        llmhub.NewAssistantMessage(llmhub.ToolCall(call.ID, call.Name, call.Arguments)),
        llmhub.NewToolResultMessage(call.ID, call.Name, llmhub.Text(result)),
    )
}

finalResp, _ := client.Generate(ctx, messages)
fmt.Println(finalResp.Text())

Tool choice helpers include AutoToolChoice, NoToolChoice, RequiredToolChoice, and NamedToolChoice("tool_name") for providers that expose tool-choice controls. Streaming tool calls are exposed on StreamChunk.ToolCalls. Parallel OpenAI-style argument fragments include ToolCallContent.Index so callers can reassemble concurrent tool calls.

Provider	Tool Calling Support
OpenAI	✅ Chat Completions tools
Anthropic	✅ Messages API tools
Gemini	✅ Function declarations
Ollama	✅ Native `/api/chat` tools
xAI	✅ Chat Completions tools
OpenRouter	✅ Chat Completions tools
Z.AI	✅ Chat Completions tools

Provider Registry

Add your own provider in a different module:

func init() {
    llmhub.MustRegisterProvider("my-llm", func(apiKey string, opts ...llmhub.Option) (llmhub.Provider, error) {
        return newMyClient(apiKey, opts...) // implement llmhub.Provider
    })
}

At runtime, consumers simply call llmhub.New("my-llm", "token").

Built-in Providers

Provider	Status	Notes
OpenAI	✅ Production	Chat Completions, multi-modal prompts, SSE streaming.
Anthropic	✅ Production	Claude 3 Messages API with streaming deltas.
Gemini	✅ Production	Gemini 1.5 multi-modal text+vision APIs, streaming JSON.
Ollama	✅ Production	Local inference via `/api/chat`, streaming friendly.
xAI	✅ Production	Grok models, API key or OAuth device flow, SSE streaming.
OpenRouter	✅ Production	OpenAI-compatible gateway to many vendors, returns per-request cost.
Z.AI	✅ Production	GLM Chat Completions; exact versioned base URLs, no forced `/v1`.

OpenAI Provider Details

Automatic /v1 suffix: When a custom base URL is provided (via WithBaseURL), the OpenAI provider appends /v1 unless the path already ends in a version segment (/v1, /v4, …). So https://api.openai.com and https://api.openai.com/v1 behave identically, and a gateway that already pins a version (e.g. https://api.z.ai/api/paas/v4) is left unchanged.

"default" model: When the model is set to "default" (case-insensitive), the provider queries the /v1/models endpoint at initialization and automatically selects the first available model. This is especially useful for self-hosted OpenAI-compatible servers (e.g. Ollama, vLLM, LocalAI) where you may not know the model name in advance:

client, err := llmhub.New("openai", "key",
    llmhub.WithBaseURL("http://localhost:11434"),
    llmhub.WithModel("default"),
)
// The provider will query http://localhost:11434/v1/models and use the first model.

xAI (Grok) Provider Details

Import providers/xai to register the xai provider. It uses the OpenAI-compatible Chat Completions API, supports streaming and tool calling, and defaults to grok-4.6. Override the model when your account has access to a different Grok model.

For xAI API keys, use it exactly like any other API-key provider—OAuth setup is not involved:

import (
    "github.com/smhanov/llmhub"
    _ "github.com/smhanov/llmhub/providers/xai"
)

// Using an API key:
client, err := llmhub.New("xai", apiKey,
    llmhub.WithModel("grok-4.6"),
)

if err != nil {
    panic(err)
}

OpenRouter Provider Details

Import providers/openrouter to register the openrouter provider. It uses OpenRouter's OpenAI-compatible Chat Completions API (https://openrouter.ai/api/v1), supports streaming and tool calling, and can route to any model available on OpenRouter using its vendor/model identifiers (e.g. x-ai/grok-4.5, openai/gpt-4o-mini):

import (
    "github.com/smhanov/llmhub"
    _ "github.com/smhanov/llmhub/providers/openrouter"
)

client, err := llmhub.New("openrouter", apiKey,
    llmhub.WithModel("x-ai/grok-4.5"),
)

OpenRouter returns the actual per-request cost in every response (usage.cost), so Response.Usage.Cost is always the exact amount charged— no WithCost rates are needed for OpenRouter.

Z.AI Provider Details

Import providers/zai to register the zai provider. It uses Z.AI's OpenAI-compatible Chat Completions API and keeps the versioned base URL exactly as supplied (no automatic /v1 suffix). New defaults to the general endpoint; NewCodingPlan uses the Coding Plan endpoint.

import (
    "github.com/smhanov/llmhub"
    "github.com/smhanov/llmhub/providers/zai"
)

client, err := llmhub.New("zai", apiKey,
    llmhub.WithBaseURL(zai.CodingPlanBaseURL),
    llmhub.WithModel("glm-5.3"),
)

To reduce binary size, providers self-register when imported, enabling tree-shaking when unused.

import (
    _ "github.com/smhanov/llmhub/providers/openai"
    _ "github.com/smhanov/llmhub/providers/anthropic"
    _ "github.com/smhanov/llmhub/providers/gemini"
    _ "github.com/smhanov/llmhub/providers/ollama"
    _ "github.com/smhanov/llmhub/providers/openrouter"
    _ "github.com/smhanov/llmhub/providers/xai"
    _ "github.com/smhanov/llmhub/providers/zai"
)

Each provider reads the shared functional options:

WithAPIKey – supply SaaS credentials (openai, anthropic, gemini, xai, openrouter, zai).
WithTokenSource – supply an OAuth / dynamic token source (xai).
WithBaseURL – point to proxies/self-hosted gateways.
WithModel, WithTemperature – customize LLM behavior per call. WithTemperature is unset by default: no temperature is sent and the provider's own default applies — often it is best to keep it that way. Calling WithTemperature(x) sends x verbatim, including 0. Note that some models (e.g. recent Anthropic ones) reject the field entirely, so omit WithTemperature when targeting them.
WithMaxTokens – only set this when you truly need a hard output cap; otherwise leave it unset to reduce the risk of truncated responses.
WithWebSearch – enable web search/grounding (Gemini: google_search tool).
WithTools, WithToolChoice – enable user-defined tool calling.
WithResponseModalities – control output modalities (e.g. "IMAGE" for Gemini image generation).
WithCost – set per-million-token pricing for cost accounting.
WithRetryOnStatus – override HTTP retry for a status (WithRetryOnStatus(429, false) surfaces rate limits immediately).
WithPromptCache – turn prompt caching on/off (enabled by default; see Prompt Caching).
WithCacheKey – set a stable per-conversation prompt cache key (see Prompt Caching).

Warning

Prefer not to use WithMaxTokens in normal application code. Provider defaults usually produce more complete answers, while an explicit cap that is too low commonly causes cut-off output.

Image Generation / Output Modalities

Gemini image-generation models (e.g. gemini-2.5-flash-image) can return images instead of—or alongside—text. Use WithResponseModalities to tell the model which output types you want:

import (
    "github.com/smhanov/llmhub"
    _ "github.com/smhanov/llmhub/providers/gemini"
)

client, _ := llmhub.New("gemini", apiKey,
    llmhub.WithModel("gemini-2.5-flash-image"),
    llmhub.WithResponseModalities("IMAGE"),
)

prompt := []*llmhub.Message{
    llmhub.NewUserMessage(
        llmhub.Text("Upscale this image to 800 pixels wide."),
        llmhub.Image("data:image/jpeg;base64,/9j/4AAQ..."),
    ),
}

resp, _ := client.Generate(ctx, prompt)

for _, part := range resp.Content {
    if img, ok := part.(*llmhub.ImageContent); ok {
        // img.URL is a data URL: "data:image/png;base64,..."
        fmt.Println("Got image:", len(img.URL), "bytes")
    }
}

Pass "TEXT" and "IMAGE" together to allow mixed text+image output:

llmhub.WithResponseModalities("TEXT", "IMAGE")

Provider	Image Output Support
Gemini	✅ Via `WithResponseModalities("IMAGE")`
OpenAI	❌ Use the Images API directly
Anthropic	❌ Not supported
Ollama	❌ Not supported
OpenRouter	❌ Not supported

Cost Accounting

llmhub can track the estimated cost of each request based on token usage and configured per-million-token rates. Costs are expressed in US dollars per 1 million tokens, matching standard LLM provider pricing.

client, _ := llmhub.New("openai", apiKey,
    llmhub.WithModel("gpt-4o"),
    llmhub.WithCost(2.50, 10.00), // $2.50 per 1M input, $10.00 per 1M output tokens
)

resp, _ := client.Generate(ctx, prompt)
fmt.Printf("Tokens: %d in, %d out\n",
    resp.Usage.PromptTokens, resp.Usage.CompletionTokens)
fmt.Printf("Cost: $%.6f\n", resp.Usage.Cost)

Cost is computed automatically after each Generate call:

$$ \text{Cost} = \frac{\text{PromptTokens} \times \text{InputRate}}{1{,}000{,}000}

\frac{\text{CompletionTokens} \times \text{OutputRate}}{1{,}000{,}000} $$

If the LLM provider returns cost directly in its API response (e.g., OpenRouter's usage.cost), that cost takes precedence and overrides any rates configured via WithCost. This also applies to OpenAI-compatible gateways and self-hosted servers that report cost or total_cost in the usage block. When no provider cost is returned and no cost rates are configured, Usage.Cost will be zero.

You can also override cost rates on a per-request basis:

// Use cheaper rates for a specific call
resp, _ := client.Generate(ctx, prompt,
    llmhub.WithCost(0.15, 0.60),
)

Latency & Timing Telemetry

llmhub measures standard latency metrics itself, so consumer applications do not need hand-rolled timestamping and chunk-tracking logic. Every streaming chunk carries a Timing snapshot, and the terminal chunk (Done or Err) carries the complete set:

stream, err := client.Stream(ctx, prompt)
if err != nil {
    log.Fatal(err)
}
for chunk := range stream {
    if chunk.Err != nil {
        log.Printf("stream error: %v", chunk.Err)
        break
    }
    fmt.Print(chunk.Delta)
    if chunk.Done {
        t := chunk.Timing
        fmt.Printf("\nconnected: %v, first byte: %v, first token: %v, total: %v (%d chunks)\n",
            t.Connected, t.FirstByte, t.FirstToken, t.Total, t.Chunks)
    }
}

Field	Meaning
`StartedAt`	When `Stream`/`Generate` was invoked
`Connected`	Time to establish the HTTP connection (via `httptrace`), including TLS; near zero on reused connections; zero when the transport did not report it
`FirstByte`	Time to the upstream's first response byte (response headers). When the request was retried (e.g. 429 backoff), it reflects the successful attempt
`FirstToken`	Time To First Token (TTFT): first content-bearing chunk (text, reasoning, or tool-call delta). Usage-only trailing frames do not count
`Total`	Full stream duration; set on the terminal chunk and on `Generate` responses, zero on earlier chunks
`Chunks`	Content-bearing chunks observed so far; final on the terminal chunk

All durations are measured from StartedAt; a zero duration always means "not available", never "instant". Non-streaming Generate returns Response.Timing with Connected, FirstByte, and Total filled. Timing is populated by the Client wrapper for every built-in provider — call a Provider directly (bypassing Client) and it stays nil.

Prompt Caching

Prompt caching is enabled by default. llmhub emits whatever top-level cache-enabling fields the destination provider understands, so repeated prompts (system messages, tool definitions, long conversation history) are served from the provider cache at a steep discount. Cache reads show up in Usage.CacheReadTokens / Usage.CacheCreationTokens on every response.

Provider	Default behavior	What llmhub sends
OpenAI	Automatic caching, billed; disabled by explicit opt-out	`prompt_cache_key` when `WithCacheKey` is set (routing hint); `prompt_cache_options {"mode":"explicit"}` when opted out
Anthropic	Automatic caching via ephemeral marker	Top-level `cache_control: {"type":"ephemeral"}` — the system places/advances breakpoints
xAI (Grok)	Automatic caching	`x-grok-conv-id` request header when `WithCacheKey` is set (plus the harmless `prompt_cache_key` body field)
OpenRouter	Vendor-dependent	`prompt_cache_key` when `WithCacheKey` is set — OpenRouter pins sticky provider routing to it from request 1
Z.AI	Automatic server-side caching	`prompt_cache_key` body field when `WithCacheKey` is set (tolerated)
Gemini	Implicit caching (2.5+ models)	Nothing to send; `WithPromptCache(false)` is a documented no-op on this lane
Ollama	Local inference caching	Nothing to send; `WithPromptCache(false)` is a no-op

Opting Out

Pass WithPromptCache(false) to disable caching on every lane:

client, _ := llmhub.New("anthropic", apiKey,
    llmhub.WithModel("claude-3-haiku-20240307"),
    llmhub.WithPromptCache(false), // no cache_control, no prompt_cache_key
)

Opting out is consistent across providers:

Anthropic: the automatic-caching marker is omitted, so nothing is cached.
OpenAI: the native provider renders prompt_cache_options {"mode":"explicit"}, which turns off OpenAI's server-side automatic caching for the request. This matters for billing: OpenAI charges cache writes at 1.25x input price on GPT-5.6+ models whether or not the cached prefix is ever read again, so a prompt of 1024+ tokens sent once costs more with caching on than off.

client, _ := llmhub.New("openai", apiKey,
    llmhub.WithModel("gpt-5.6"),
    llmhub.WithPromptCache(false), // sends prompt_cache_options {"mode":"explicit"}
)

Warning

Cache writes cost extra: Anthropic bills 1.25x on written tokens (5-minute TTL; 2x with the 1-hour TTL) and OpenAI bills 1.25x on GPT-5.6+ even for automatic caching, while cache reads run 0.1x–0.5x of the normal input price. One-shot workloads that send large unique prompts once and never reuse them should pass WithPromptCache(false). Multi-turn agents should keep caching on.

When routing to OpenAI models through OpenRouter, the openrouter provider does not emit OpenAI's control (it is OpenAI-specific vocabulary). Use the WithExtraBody escape hatch there — an explicit prompt_cache_options always wins over anything llmhub generates:

llmhub.WithExtraBody(map[string]json.RawMessage{
    "prompt_cache_options": json.RawMessage(`{"mode":"explicit"}`),
})

Cache Keys

WithCacheKey supplies a stable per-conversation key for providers that route cached requests:

client, _ := llmhub.New("openrouter", apiKey,
    llmhub.WithModel("x-ai/grok-4.5"),
    llmhub.WithCacheKey("conversation-123"), // sticky routing from request 1
)

On OpenRouter this pins multi-turn agents to the same provider from the first request of a conversation; on OpenAI it improves routing to the same cache shard; on xAI it is also sent as the x-grok-conv-id header. Leave it unset to rely on provider-automatic caching (OpenRouter falls back to message-hash sticky routing after the first successful request).

Web Search / Grounding

Some providers support web search to ground responses in real-time information:

client, _ := llmhub.New("gemini", apiKey,
    llmhub.WithModel("gemini-2.5-flash"),
    llmhub.WithWebSearch(true),
)

prompt := []*llmhub.Message{
    llmhub.NewUserMessage(llmhub.Text("What are the latest news about Apple Inc?")),
}

resp, _ := client.Generate(ctx, prompt)
fmt.Println(resp.Text())

Provider	Web Search Support
Gemini	✅ Uses `google_search` tool
OpenAI	❌ Not supported
Anthropic	❌ Not supported
Ollama	❌ Not supported
xAI	❌ Not supported
OpenRouter	❌ Not supported

OAuth Token Sources

OAuth is opt-in and provider-dependent. It is not required by existing API-key providers. Currently, xai is the built-in provider that accepts WithTokenSource; future OAuth providers can reuse the generic auth and auth/oauth2 packages.

OAuth login is always explicit and application-controlled: llmhub.New, Generate, and Stream will never open a browser or wait for a user to sign in.

Using an Existing Token File

import (
    "github.com/smhanov/llmhub"
    "github.com/smhanov/llmhub/providers/xai"
)

// Discover an existing ~/.grok/auth.json, ~/.xgroxy/auth.json, or XAI_AUTH_FILE.
authPath := xai.DefaultAuthPath("")
store := xai.NewFileTokenStore(authPath)
source := xai.NewTokenSource(store)

client, err := llmhub.New("xai", "",
    llmhub.WithTokenSource(source),
    llmhub.WithModel("grok-4.6"),
)
if err != nil {
    panic(err)
}

WithTokenSource takes precedence over the positional apiKey argument and WithAPIKey. If neither one is supplied to xai, construction fails with llmhub.ErrInvalidInput.

Performing an Interactive Device Login

Login is an explicit, application-controlled operation that never blocks inside library constructors:

import (
    "context"
    "fmt"

    "github.com/smhanov/llmhub/providers/xai"
)

func login(ctx context.Context) error {
    store := xai.NewFileTokenStore(xai.DefaultAuthPath(""))
    flow := xai.NewDeviceFlow()

    authz, err := flow.Start(ctx)
    if err != nil {
        return err
    }

    fmt.Printf("Please visit: %s\n", authz.VerificationURI)
    if authz.VerificationURIComplete != "" {
        fmt.Printf("Or open: %s\n", authz.VerificationURIComplete)
    }
    fmt.Printf("And enter code: %s\n", authz.UserCode)

    token, err := flow.Wait(ctx, authz)
    if err != nil {
        return err
    }

    return store.Save(ctx, token)
}

Credential Precedence & Error Handling

Precedence: WithTokenSource takes precedence over WithAPIKey and the positional apiKey argument.
Refresh: xai.NewTokenSource refreshes access tokens before they expire and performs one refresh/retry after an API 401 response. It safely handles rotating refresh tokens within one process.
Reauthentication: If a refresh token is expired or revoked, provider calls return an error wrapping auth.ErrReauthenticationRequired. Use errors.Is to prompt for a new interactive login:
```
if errors.Is(err, auth.ErrReauthenticationRequired) {
    // Start the device flow again and save the new token.
}
```
Security: New token directories/files are created with private 0700/0600 permissions on Unix. Do not print, commit, or pass access/refresh tokens as command-line arguments.

xAI OAuth acceptance example

examples/xai-oauth is a real end-to-end example. It performs an explicit device login when needed, can force a token refresh, then verifies both a non-streaming and a streaming Grok response. It requires an authorized xAI account and intentionally is not part of the offline test suite.

go run ./examples/xai-oauth \
  -auth-file /private/path/auth.json \
  -force-login \
  -verify-refresh \
  -model grok-4.6

The command prints a verification URL and one-time code, waits for you to authorize it in a browser, and succeeds only after it prints:

refresh: OK
generate: OK
stream: OK
xai oauth acceptance: PASS

Use a dedicated private auth-file path for this check. The example never prints access or refresh tokens.

Need multi-provider routing? Instantiate one llmhub.Client per provider and switch at runtime:

openaiClient := llmhub.MustNew("openai", os.Getenv("OPENAI_API_KEY"), llmhub.WithModel("gpt-4o"))
claudeClient := llmhub.MustNew("anthropic", os.Getenv("ANTHROPIC_API_KEY"), llmhub.WithModel("claude-3-opus-20240229"))

func answer(ctx context.Context, prompt []*llmhub.Message, vendor string) (*llmhub.Response, error) {
    switch vendor {
    case "anthropic":
        return claudeClient.Generate(ctx, prompt)
    default:
        return openaiClient.Generate(ctx, prompt)
    }
}

Testing

go test ./...

CLI Test Tool

A command-line tool is included for end-to-end testing of providers. Build and run it from the repository root:

go run ./examples/cli [options]

Options

Flag	Description
`-provider`	Provider name: `openai`, `anthropic`, `gemini`, `ollama`, `xai`, `openrouter` (required)
`-model`	Model identifier (e.g., `gpt-4o`, `claude-3-haiku-20240307`, `gemini-2.5-flash`, `grok-4.6`, `x-ai/grok-4.5`)
`-api-key`	API key (or use env vars `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, `XAI_API_KEY`, `OPENROUTER_API_KEY`)
`-auth-file`	Path to token file for OAuth providers (e.g. xAI)
`-base-url`	Override provider base URL (useful for Ollama or proxies)
`-prompt`	Text prompt to send
`-prompt-file`	File containing the prompt text
`-images`	Comma-separated list of image file paths or URLs
`-stream`	Enable streaming mode
`-temperature`	Sampling temperature (default: 0.7)
`-max-tokens`	Hard cap on generated tokens; leave unset unless needed to avoid truncation
`-no-cache`	Disable prompt caching (caching is enabled by default)
`-cache-key`	Stable per-conversation cache key (`prompt_cache_key`; `x-grok-conv-id` header on xAI)
`-input-cost`	Cost per 1M input tokens in USD (for cost accounting)
`-output-cost`	Cost per 1M output tokens in USD (for cost accounting)
`-timeout`	Request timeout duration (e.g. `30s`, `2m`, `10m`)

Examples

Text generation with Ollama (self-hosted):

go run ./examples/cli \
  -provider ollama \
  -model qwen3:32b \
  -base-url https://ollama.example.com \
  -prompt "Why is the sky blue?"

Text generation with Gemini:

go run ./examples/cli \
  -provider gemini \
  -model gemini-2.5-flash \
  -api-key YOUR_GEMINI_KEY \
  -prompt "Explain quantum entanglement simply."

Vision/image input with Gemini:

go run ./examples/cli \
  -provider gemini \
  -model gemini-2.5-flash \
  -api-key YOUR_GEMINI_KEY \
  -prompt "Describe this image in detail." \
  -images cat.jpg

Streaming mode with OpenAI:

go run ./examples/cli \
  -provider openai \
  -model gpt-4o \
  -api-key YOUR_OPENAI_KEY \
  -prompt "Write a haiku about coding." \
  -stream

OpenRouter with provider-reported cost:

go run ./examples/cli \
  -provider openrouter \
  -model x-ai/grok-4.5 \
  -api-key YOUR_OPENROUTER_KEY \
  -prompt "Reply with exactly: OK"
# Tokens and the exact Cost charged by OpenRouter are printed automatically.

Using environment variables:

export OPENAI_API_KEY=sk-...
go run ./examples/cli -provider openai -model gpt-4o -prompt "Hello!"

Using an existing xAI OAuth token file:

go run ./examples/cli \
  -provider xai \
  -auth-file ~/.grok/auth.json \
  -model grok-4.6 \
  -prompt "Reply with exactly: OK"

With cost accounting:

go run ./examples/cli \
  -provider openai \
  -model gpt-4o \
  -input-cost 2.50 \
  -output-cost 10.00 \
  -prompt "Explain Go interfaces."

Contributing

Issues and PRs are welcome! Start by filing an issue describing the provider or feature you would like to add, then open a PR with tests and documentation. Check the existing provider stubs (Anthropic, Gemini, Ollama) for extension points.

This project uses a rebase-only workflow for merging into main: PRs are rebased onto main and merged with a fast-forward rebase (gh pr merge --rebase) — never merge commits or squash. Keep your branches up to date with git rebase main and keep commits small and clean, as they land in main individually.

Name		Name	Last commit message	Last commit date
auth		auth
examples		examples
internal		internal
providers		providers
.gitignore		.gitignore
AGENTS.md		AGENTS.md
LICENSE		LICENSE
README.md		README.md
client.go		client.go
client_test.go		client_test.go
errors.go		errors.go
go.mod		go.mod
options.go		options.go
options_test.go		options_test.go
provider.go		provider.go
provider_test.go		provider_test.go
registry.go		registry.go
registry_test.go		registry_test.go
timing.go		timing.go
timing_test.go		timing_test.go
types.go		types.go
types_test.go		types_test.go

Latest commit

History

Folders and files

Repository files navigation