Serve gemma-4-31b-4bit as flagship (fully resident) + fleet fixes; DeepSeek-V4 machinery parked in-tree

[anupsv]

Summary

Scope pivot (2026-07-05): the flagship serving target is now mlx-community/gemma-4-31b-4bit (18.4GB dense, fully resident) — DeepSeek-V4 is no longer the rollout target. The DSV4 machinery below stays in-tree, tested, and inert for non-DSV4 models (expert streaming, sequential serving, and DSML are all opt-in per model). Gemma 4 31B needs none of it: reference docs/reference/gemma-4-31b-serving.md, fleet admission tests coordinator/registry/gemma4_31b_resident_test.go (registration: default size, min_ram_gb=32).

What this PR ships, by relevance to the new flagship:

Applies to Gemma 4 31B (and the whole fleet):

• NAX release posture: JIT-compiled NAX bf16 GEMM kernels are nondeterministic on M5 for ALL models (root-caused: AOT-compiled identical kernel source is clean; mlx-swift compiles jit_kernels.cpp) — releases build with -Xcc -DMLX_METAL_NO_NAX. See docs/spikes/nax-nondeterminism-m5.md + validated upstream issue draft.
• Qwen3.5-VLM stale ropeDeltas crash fix (E2E flake on master: provider WebSocket disconnect under concurrent load → all requests 502 #513, mlx-swift-lm#65): provider died under concurrent load; fixed + regression test. MLXVLM Gemma4 audited for the same pattern — clean. This unblocked E2E CI fleet-wide.
• Fleet catalog catalog_size_gb override (coordinator/api/catalog_size_override.go): registration-side correction for any model whose resident footprint differs from its on-disk size. Not needed by Gemma 4 31B (resident = on-disk), required by streaming models.
• Cold-start expert-cache warming, capacity/cooldown merges from master, serving docs.

DSV4-specific (parked, in-tree, tested):

• MoE expert SSD streaming (byte-budgeted LRU, UnifiedMemoryCap-derived budgets, purge-on-unload, usage-profile warming)
• Sequential serving route for models the batched engine can't represent
• Native DeepSeek prompt encoding + DSML tool-call parser
• DeepSeek-V4 model port correctness fixes in mlx-swift-lm (submodule)

Before / after

flowchart TB
  subgraph Before["Before: gemma-4-31b request"]
    A1[POST /v1/chat/completions] --> B1["model not in catalog / provider crash risk under concurrent load (#513)"]
  end
  subgraph After["After: gemma-4-31b request (32–128GB box)"]
    A2[POST /v1/chat/completions] --> B2["catalog: size_gb=18.4 (default derivation), min_ram_gb=32"]
    B2 --> C2["admission → load_model → fully-resident load via VLMModelFactory (Gemma4)"]
    C2 --> D2["continuous batching (RotatingKVCache) → ChatML-injected template → SSE"]
  end

Loading

flowchart TB
  subgraph BeforeCode["Before: code"]
    E1["Qwen35.swift decode: repeated(ropeDeltas, batchSize) on stale state → broadcast trap, provider dies"]
    F1["SyncModelCatalog: SizeGB always raw on-disk total"]
  end
  subgraph AfterCode["After: code"]
    E2["Qwen35.swift: shape-total guard (ndim==1 && dim(0)==batchSize) + debug assert — mlx-swift-lm#65"]
    F2["catalogSizeGBForRow: runtime_parameters.catalog_size_gb override when present, else unchanged"]
    G2["gemma4_31b_resident_test.go: admission pinned across 32–128GB tiers with default registration"]
  end

Loading

Validation

• Coordinator: go test ./registry/ -run TestGemma31b — admission across 32/36/48/64/96/128GB, 24GB rejection, cold-spill at floor. Full go test ./... green.
• mlx-swift-lm: Gemma4 suites green; Qwen35 regression test (mlx-swift-lm#65) green; DSV4/streaming/tool suites unchanged and green.
• Live on-box (M5 Max) Gemma 4 31B validation: load, single + batched decode, SSE, mid-decode admission waves — in progress, results will be appended.
• (DSV4 historical validation retained in docs/reference/deepseek-v4-serving.md.)

Test plan

• CI green (E2E unblocked by the E2E flake on master: provider WebSocket disconnect under concurrent load → all requests 502 #513 fix — first green run on this PR after the cherry-pick)
• On-box Gemma 4 31B live validation numbers appended
• Rollout (separate from this PR): registry publish for gemma-4-31b-4bit (min_ram_gb=32, no size override), pricing entry, provider release

Made with Cursor

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
d-inference	Ready	Preview	Jul 6, 2026 6:30am
d-inference-console-ui-dev	Ready	Preview	Jul 6, 2026 6:30am
d-inference-landing	Ready	Preview	Jul 6, 2026 6:30am

[github-actions]

This PR adds DeepSeek-V4 sequential-serving and MoE expert SSD-streaming support; it is primarily a feature diff with no meaningful change to the security posture — no mitigations are weakened or removed.

---

Trust boundaries touched

• TB-002 (Coordinator → Provider WebSocket) — server.go catalog size helper change
• TB-003 (Provider operator vs process) — ProviderConfig.swift new config fields, BatchScheduler*.swift inference path changes
• TB-007 (Provider inference engine) — entire BatchScheduler family
• TB-009 (Apple attestation chain) — indirectly, via BatchScheduler.swift weight-hash reset path

---

Per-threat assessment

T-007 / T-012 / T-027 — Weight hash and model-integrity enforcement

ℹ️ Neutral on the weight-hash enforcement gap (SEC-007 remains open).

• BatchScheduler+ModelLifecycle.swift: weightHash is still passed through loadModel and stored in currentWeightHash unchanged. The new expertStreamingCacheBytes / expertStreamingConfigured fields are reset in stopCurrentEngine (BatchScheduler.swift lines ~608–610), which also resets currentWeightHash — no regression there.
• DeepseekV4TemplateFix.swift: model-type detection uses modelId.lowercased().replacingOccurrences(of: "-", with: "_").contains("deepseek_v4"). A model registered under a name that doesn't contain this substring will silently fall through to applyChatTemplate, which for a model with no Jinja template throws missingChatTemplate — that's a correct fail-loud path, not a silent bypass.
• No change to the coordinator-side weight-hash admission logic; SEC-007 (fail-open on omitted hash) is untouched.

T-028 — Residual inference data in GPU memory

ℹ️ Neutral. The expert-streaming cache purge in stopCurrentEngine (BatchScheduler.swift ~+525–535) is a welcome addition: it releases the process-wide DeepseekV4ExpertStreaming.cache after all in-flight work is fenced. This reduces the window in which expert weight data sits in MLX cache memory between model unloads, but it does not address the open finding about GPU-buffer zeroing between consumer requests (the existing limitation for KV/activation buffers). No regression.

T-041 — Cross-tenant prefix-cache sharing / TTFT timing oracle

ℹ️ Neutral. BatchScheduler.swift changes don't touch prefixCacheManager or EncryptedPrefixCachePersistence. The new requiresSequentialServing path (effectiveMaxConcurrentRequests returns 1 for DeepSeek-V4) means prefix-cache sharing across concurrent tenants cannot occur for this model by construction — a minor implicit improvement, but not a code-level mitigation of SEC-035.

T-004 / T-024 — Admin/release key validation (server.go)

ℹ️ Neutral. The only change is SizeGB: catalogSizeGBForRow(row) replacing an inline expression. No auth, middleware, or key-comparison logic is touched.

T-021 / T-040 — MDM webhook / host-header injection (server.go)

ℹ️ Neutral. Same single-line catalog-size change; no webhook or URL-generation code is modified.

T-011 — X25519 key in config file (ProviderConfig.swift)

ℹ️ Neutral. Two new fields (streamExperts: Bool, expertCacheGb: Double) are added to BackendSettings. Neither is a secret; both use decodeIfPresent with safe defaults. No change to key storage or the existing config-file attack surface.

---

New attack surface not covered by existing threats

DeepseekV4TemplateFix string-match model dispatch (DeepseekV4TemplateFix.swift, BatchScheduler+Submit.swift ~L393–406, MultiModelBatchSchedulerEngine.swift ~L388–401)

The isDeepseekV4Hint check normalises hyphens to underscores and matches on substring "deepseek_v4" in either modelType or modelId. A malicious provider operator who registers a model with an ID containing "deepseek_v4" (e.g. "my-deepseek_v4-spoofed") would route all its prompts through DeepseekV4Encoding.encode instead of applyChatTemplate. This affects the provider-operator-vs-process boundary (TB-003): the operator controls model IDs in provider.toml and the local HuggingFace cache directory. The impact is prompt mis-formatting (potentially gibberish output or prompt injection via the encoding seam) for that provider's consumers — consistent with ADV-001's ability to serve manipulated outputs (T-007/T-012), but via a new code path. This is not a new class of threat, but the dispatch heuristic is fragile; a strict model_type == "deepseek_v4" equality check on the coordinator-verified catalog entry would be more appropriate than a substring match on operator-controlled strings.

Sequential-admission rejection leaks model identity (BatchScheduler+Submit.swift ~L100–109)

The error string "token_budget_exhausted: sequential-serving model is busy; retry shortly" is distinct from the standard token-budget error. If this string is forwarded to consumers via the coordinator's SSE error path, it leaks that the serving provider is running a sequential-serving model (i.e. DeepSeek-V4). This is a minor information-disclosure issue across TB-001 / TB-002. Worth checking that provider.go's inference error relay either redacts or normalises provider-side error strings before forwarding to consumers.

---

Open findings resolved by this PR

None. SEC-007, SEC-016, SEC-017, SEC-035 and other open findings are all unaffected.

---

🔐 Threat model: docs/threat-model.yaml · Updates on each push to this PR

[blacksmith-sh]

Found 2 test failures on Blacksmith runners:

Failures

Test	View Logs
github.com/eigeninference/d-inference/e2e/TestIntegration_ConcurrentRequests	View Logs
github.com/eigeninference/d-inference/e2e/TestProfile_SingleProviderNonStreaming	View Logs

^{Need help on this PR? Tag /codesmith with what you need.}