provider: cacheFormat "anthropic" puts cache_control inside tool message content, gateway ignores it, no caching after tool results

Summary

With an openai-compatible provider pointing at an Anthropic-backed gateway and compatibility.cacheFormat: "anthropic", V2 puts the cache_control marker inside the tool message's content array. The gateway ignores it there, so no caching happens whenever the last message is a tool result. In long tool-heavy sessions every step re-bills the whole conversation as uncached input.

Environment

opencode version: 2.0.32
OS: Darwin 25.6.0 (macOS, arm64)
Terminal: iTerm.app, TERM=xterm-256color
Shell: /bin/zsh
Install/channel: latest, curl install
Active plugins: opencode-beads (ruled out, same behaviour with and without it)

Reproduction

"providers": {
  "gateway": {
    "package": "aisdk:@ai-sdk/openai-compatible",
    "settings": { "baseURL": "https://<gateway>/openai/", "apiKey": "{env:KEY}" },
    "models": {
      "claude-sonnet-5-5": { "compatibility": { "cacheFormat": "anthropic" } }
    }
  }
}

Start a session on that model.
Ask it to read a large file in chunks, one read call per turn.
Watch tokens.cache.read / tokens.cache.write per step, or log request bodies with a proxy.

Expected Behavior

Each step reads nearly the whole previous conversation from cache and writes only the new tail.

Actual Behavior

Once the last message is a tool result, cache read stays at the system prompt + tools size (11,331 tokens) and cache write is 0, while the prompt grows:

Step	Prompt tokens	Cache read
4	96,315	11,331
6	146,218	11,331
11	272,176	11,331

Captured request bodies have cache_control: {"type":"ephemeral"} on the system message, the last tool definition, and messages[N].content[0] of the final tool message. Ignoring cache_control, each request is identical to the previous one up to its newest messages, so the prefix is stable.

Sending the same ~123K-token request twice straight to the gateway with the marker in different places:

Marker position	2nd request cache read
on a `user` message content part	123K (works)
on the `tool` message itself (`{"role":"tool","content":"...","cache_control":{...}}`)	123K (works)
inside a `tool` message's `content` array (what V2 sends)	0
on an `assistant` message	0

Additional Context

Likely regression: before moving to V2 (V1-format config, no compatibility) the same gateway gave an 86 to 97% hit rate. I did not verify which V1 build that was.
Suggested fix: for cacheFormat: "anthropic" on openai-compatible providers, put the marker at message level for tool messages.
Workaround confirmed: a plugin on http.request that moves cache_control from tool message content parts to the tool message. With it, the same replay read 73K, 96K, 123K and 146K tokens from cache on steps 4 to 7.

await ctx.session.hook("http.request", async (event) => {
  const req = event.request
  if (req.method !== "POST" || !req.url.includes("/chat/completions")) return
  const body = JSON.parse(await req.clone().text())
  let changed = false
  for (const m of body.messages ?? []) {
    if (m.role !== "tool" || !Array.isArray(m.content)) continue
    for (const p of m.content) {
      if (p?.cache_control) { m.cache_control = p.cache_control; delete p.cache_control; changed = true }
    }
  }
  if (!changed) return
  const headers = new Headers(req.headers); headers.delete("content-length")
  event.request = new Request(req.url, { method: req.method, headers, body: JSON.stringify(body) })
})

Related but different: setCacheKey sends promptCacheKey (wrong) instead of cache_control on content blocks for openai-compatible Bedrock proxies (Bifrost, LiteLLM) #25984 / PR fix: inject cache_control on content blocks for openai-compatible proxies to Anthropic backends (Bifrost, LiteLLM, Databricks) #25985 (setCacheKey vs cache_control on openai-compatible Anthropic proxies).

Thanks for the detailed report and the side-by-side gateway tests.

The code matches your capture: with cacheFormat: "anthropic" on an openai-compatible model, OpenCode always puts cache_control on a text part inside the tool message's content array, never on the tool message itself. That placement is intentional in the code, and it's the same in 2.0.32 and on the current dev.

I can't confirm the gateway side here. Whether a gateway honours the marker inside tool content or only at message level depends on the gateway, and testing it needs a real Anthropic-backed gateway with caching. I don't have access to one, so a maintainer will need to check this, ideally against several common gateways.

It would help to know which gateway you're using (e.g. LiteLLM, Bifrost, a vendor's gateway) and its version, if you can share that. Your http.request plugin is a good workaround in the meantime. Related: #25984 (different issue: promptCacheKey vs cache_control) and #54945 (where cacheFormat was added).

Which gateway (product and version) is behind https://<gateway>/openai/? For example LiteLLM, Bifrost, Portkey, Cloudflare AI Gateway, or an internal/vendor gateway.

We checked the current placement against real Anthropic traffic (Claude Haiku 5.5, a request ending in a tool result, the same ~23K-token request sent twice). Both gateways cache with the marker inside the tool message's content array, which is where OpenCode puts it today:

OpenRouter: 22,864 tokens read from cache on the second request
LiteLLM 1.106: 22,862 tokens read from cache on the second request

Both also cache with the marker on the tool message itself. So the content-part placement works on these gateways, and the behaviour you're seeing looks specific to how your gateway translates OpenAI chat messages into Anthropic requests. Knowing which gateway it is lets us reproduce it and decide whether to also set the marker at message level for tool messages.

I have a reproduction and a fix ready for this: the marker for a tool result is nested in the tool message's content array, which a gateway does not translate. Moving it to the tool message for the anthropic cache format on the openai-compatible route restores the cache read. Opening a PR.

fix(ai): put a tool result's cache marker on the tool message for ant…

Here is the fix, verified against a compiled request. The branch is pushed at dat-rd/opencode:fix-tool-cache-marker; GitHub is currently refusing to open a pull request from this fork to this repository, so the patch is here in the meantime.

The marker for a tool result is nested in the tool message's content array, which is where the chat protocol puts it. A gateway that translates the chat shape into Anthropic's does not translate a marker from that position, so a tool-heavy session is billed as uncached input from the first tool result on.

With cacheFormat: "anthropic" on the openai-compatible route, the marker now goes on the tool message. OpenRouter keeps the content-part position, which is what its own contract documents, and the chat route is unchanged.

Before, with a tool result as the last message:

{ "role": "tool", "tool_call_id": "call_1", "content": [{ "type": "text", "text": "file contents", "cache_control": { "type": "ephemeral" } }] }

After:

{ "role": "tool", "tool_call_id": "call_1", "content": "file contents", "cache_control": { "type": "ephemeral" } }

The patch:

diff --git a/packages/ai/src/protocols/openai-chat.ts b/packages/ai/src/protocols/openai-chat.ts
index 4d6a58e7c2..1e9e3e5978 100644
--- a/packages/ai/src/protocols/openai-chat.ts
+++ b/packages/ai/src/protocols/openai-chat.ts
@@ -160,6 +160,7 @@ export const OpenAIChatMessage = Schema.Union([
     role: Schema.Literal("tool"),
     tool_call_id: Schema.String,
     content: Schema.Union([Schema.String, Schema.Array(OpenAIChatTextContent)]),
+    cache_control: Schema.optional(OpenAIChatCacheControl),
   }),
 ]).pipe(Schema.toTaggedUnion("role"))
 type OpenAIChatMessage = Schema.Schema.Type<typeof OpenAIChatMessage>
@@ -326,6 +327,8 @@ export interface ParserState {
 // fields into `LLMRequest`.
 interface LoweringOptions {
   readonly cacheControl?: (cache: CacheHint | undefined) => OpenAIChatCacheControl | undefined
+  /** Puts the marker for a tool result on the tool message, where an Anthropic-backed gateway reads it. */
+  readonly cacheControlAtMessageLevel?: boolean
   readonly toolCallID?: (id: string) => string
   /** Project provider-specific fields from the exact source, even when other messages are dropped during lowering. */
   readonly assistant?: (source: LLMRequest["messages"][number], message: OpenAIChatMessage) => OpenAIChatMessage
@@ -513,6 +516,7 @@ const lowerToolMessages = Effect.fnUntraced(function* (
           options.toolCallID?.(part.id) ?? part.id,
           ProviderShared.toolResultText(part),
           options.cacheControl?.(part.cache),
+          options.cacheControlAtMessageLevel === true,
         ),
       )
       continue
@@ -520,7 +524,12 @@ const lowerToolMessages = Effect.fnUntraced(function* (
     const content: ReadonlyArray<Tool.Content> = part.result.value
     const text = content.filter((item) => item.type === "text").map((item) => item.text)
     messages.push(
-      toolMessage(options.toolCallID?.(part.id) ?? part.id, text.join("\n"), options.cacheControl?.(part.cache)),
+      toolMessage(
+        options.toolCallID?.(part.id) ?? part.id,
+        text.join("\n"),
+        options.cacheControl?.(part.cache),
+        options.cacheControlAtMessageLevel === true,
+      ),
     )
     const files = content.filter((item) => item.type === "file")
     attachments.push(...(yield* Effect.forEach(files, (item) => lowerMedia(ProviderShared.toolFileMedia(item)))))
@@ -528,12 +537,28 @@ const lowerToolMessages = Effect.fnUntraced(function* (
   return { messages, attachments }
 })
 
-// Chat cache breakpoints belong on text content parts, not on the message itself.
-const toolMessage = (toolCallID: string, text: string, cacheControl: OpenAIChatCacheControl | undefined) => ({
-  role: "tool" as const,
-  tool_call_id: toolCallID,
-  content: cacheControl === undefined ? text : [{ type: "text" as const, text, cache_control: cacheControl }],
-})
+// Chat cache breakpoints belong on text content parts, not on the message itself. A gateway
+// that translates the chat shape into Anthropic's is the exception: it reads the marker for a
+// tool result off the tool message, and one nested in the content array is not a position it
+// translates, so the request is billed as uncached input from then on.
+const toolMessage = (
+  toolCallID: string,
+  text: string,
+  cacheControl: OpenAIChatCacheControl | undefined,
+  cacheControlAtMessageLevel: boolean,
+) =>
+  cacheControlAtMessageLevel
+    ? {
+        role: "tool" as const,
+        tool_call_id: toolCallID,
+        content: text,
+        ...(cacheControl === undefined ? {} : { cache_control: cacheControl }),
+      }
+    : {
+        role: "tool" as const,
+        tool_call_id: toolCallID,
+        content: cacheControl === undefined ? text : [{ type: "text" as const, text, cache_control: cacheControl }],
+      }
 
 const lowerMessage = Effect.fnUntraced(function* (
   message: OpenAIChatRequestMessage,
@@ -798,7 +823,16 @@ export const fromRequest = Effect.fn("OpenAIChat.fromRequest")(function* (
     request.model.compatibility?.zaiToolStream ?? detectZaiToolStream(provider, baseURL, request.model.id)
   const hasHistory = hasToolHistory(request.messages)
   const hasActiveTools = flattened.tools.length > 0
-  const lowering = { ...options, cacheControl: options.cacheControl ?? declaredCacheControl(request) }
+  const lowering = {
+    ...options,
+    cacheControl: options.cacheControl ?? declaredCacheControl(request),
+    // A gateway that speaks the chat shape in front of Anthropic reads a tool result's marker
+    // off the tool message, and one nested in the content array is not a position it translates.
+    // OpenRouter is left alone: it documents the content-part position for its own contract.
+    cacheControlAtMessageLevel:
+      request.model.compatibility?.cacheFormat === "anthropic" &&
+      request.model.route.id === "openai-compatible-chat",
+  }
   return {
     model: request.model.id,
     messages: yield* lowerMessages(flattened.request, lowering),

A test in packages/ai/test/provider/openai-compatible-cache.test.ts compiles a request whose last message is a tool result and reads where the marker ended up, for the anthropic format, for no declared format, and for OpenRouter. It fails on the code before this change. The full packages/ai suite is 1826 pass, 0 fail.

It's a private, self-hosted OpenAI-compatible gateway in front of Anthropic models. I don't know the product or version, so I can't say more than that.

What I can confirm from my side is that on this gateway the marker is ignored inside a tool message's content array and honoured on the tool message itself. That's the placement table in the issue. The http.request plugin that moves it fixed it end to end, and I'm running it now.

dat-rd's patch does the same thing at the source, scoped to cacheFormat: "anthropic" on the openai-compatible chat route, so it looks right to me. Happy to test a build of it against this gateway if that helps.

Navigation Menu

Repository navigation

provider: cacheFormat "anthropic" puts cache_control inside tool message content, gateway ignores it, no caching after tool results #57199

Summary

Environment

Reproduction

Expected Behavior

Actual Behavior

Additional Context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

provider: cacheFormat "anthropic" puts cache_control inside tool message content, gateway ignores it, no caching after tool results #57199

Description

Summary

Environment

Reproduction

Expected Behavior

Actual Behavior

Additional Context

Activity

opencode-agent commented on Oct 19, 2026

opencode-agent commented on Oct 19, 2026

dat-rd commented on Oct 20, 2026

dat-rd commented on Oct 20, 2026

HawkingRadiation42 commented on Oct 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions