[Web] WebGPU EP creates a new GPUBindGroup for every dispatch (~317 per decoded token); bind groups are costly in WebKit (memory held until GC) #33750

Describe the issue

Outside graph capture, the WebGPU EP creates a new bind group for every dispatch. Nothing reuses them across runs. For LLM decoding that comes to about 300 createBindGroup calls per generated token. That costs CPU on every browser. On Safari it also costs memory that can only be freed by GC, and on iOS this got our page killed.

Where in the code (same at the build we use and on current main):

• WebGpuContext::Run builds bind_buffers from the input/output tensors plus a uniform buffer, then calls CreateBindGroup(...) once per program dispatch:
  webgpu_context.cc#L704 @ 8d85527a0, #L987 @ main
• WebGpuContext::CreateBindGroup calls wgpuDeviceCreateBindGroup directly, with no lookup first:
  #L1057-L1097 @ 8d85527a0, #L1529-L1569 @ main
• The only place a bind group is kept past one dispatch is CapturedCommandInfo::bind_group in graph-capture mode, where it is replayed. Pipelines and bind group layouts are cached through the program cache, but bind groups are not.
• The uniform buffer also comes fresh from the buffer manager on every dispatch (#L674). So any cache keyed on the bound buffers would also need uniform buffers to be stable for each node, or written at a dynamic offset into a shared buffer.

Measurements. We wrapped GPUDevice.prototype.createBindGroup in the worker and counted calls. Model: daveycodez/LFM3-1.7B-Web, q4f16, graph capture off, default (bucket) buffer cache.

From one step to the next, a node usually binds the same buffers again, since the bucket cache hands back the same GPUBuffers. Even so, a new bind group is created every time.

Why this hurts on Safari / WebKit. WebGPU has no GPUBindGroup.destroy(), so a bind group's memory is freed only when its JS wrapper is garbage-collected.

• WebKit's Device::createBindGroup allocates a separate Metal argument buffer for each bind group: argumentBuffer[stage] = safeCreateBuffer(encodedLength, MTLStorageModeShared) (BindGroup.mm#L1341).
• Until WebKit commit 5185b8f77649 (Oct 15 2026, bug 326687), a bind group also kept destroyed textures and buffers it referenced alive until it was collected.
• A worker running ORT allocates very little JS memory, so GC rarely runs. Bind groups pile up: about 150,000 were uncollected when iOS killed the page at 2,048 MB on an iPhone 16 Pro Max.
• Our workaround: every 16 tokens, allocate ~32 MB of throwaway ArrayBuffers to force a collection. A FinalizationRegistry on the bind groups confirms they then get collected, and the uncollected count stays at ~250–7,000. This is a hack, and every app using ORT on Safari would need its own copy of it.

Ask: reuse bind groups outside graph capture. Possible approaches:

• Cache them keyed on (bind group layout / program key, and for each entry: buffer, offset, size). Drop entries when a buffer goes back to the allocator for good.
• Or keep the bind group for each node/program instance, and reuse it while its input/output buffers and uniform buffer come back unchanged from the cache.

This would cut ~300 API calls per token on all browsers, and on Safari it would keep this memory from building up between GCs.

To reproduce

1. In a dedicated worker, load daveycodez/LFM3-1.7B-Web with Transformers.js 4.3.2: device: "webgpu", dtype: "q4f16", executionProviders: ["webgpu"], graph capture off. We use the JSPI build (ort-wasm-simd-threaded.jspi.mjs).
2. Before loading, wrap GPUDevice.prototype.createBindGroup to count calls (and optionally register each result with a FinalizationRegistry to count collections).
3. Generate a few hundred tokens and log the count every 32 tokens: ~10k per 32 tokens.
4. On iOS Safari, a long generation keeps growing in memory until jetsam kills the page, unless GC is forced.

App with the instrumentation: https://github.com/daveycodez/lofi-agent (apps/web/src/lib/llm/local-model.worker.ts).

Urgency

Not blocking: we have a workaround (forcing GC from the worker). Without it, long generations on iPhone end with the tab being killed.

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

onnxruntime-web 1.31.0-dev.20260914-8d85527a0 (commit 8d85527). The code is unchanged on main at 74bda51.

Execution Provider

'webgpu' (WebGPU)

System

• iPhone 16 Pro Max (iPhone17,2), iOS 27.2, Safari. Page killed at 2,048 MB.
• Also measured on a Mac in Chrome.