feat(core): introduce experimental IPC v2 while preserving SDK contracts

[jrschumacher]

Goal

Introduce IPC v2 as an experimental, opt-in local transport while preserving the existing SDK-facing service contract. Compare candidate dispatch mechanisms before selecting an implementation; do not assume serialization-free direct dispatch is the right trade-off.

Services must use the same typed interfaces and request/response schemas in local and remote deployments. Business logic must not branch on IPC version. Keep Connect-based IPC v1 and remote transport supported while downstream implementations pilot reliability and performance.

Options to evaluate

Candidate	Preserves	Potentially removes	Main concern
Optimized existing IPC	SDK, Connect encoding, HTTP/2, current middleware	Selected compression/avoidable overhead	Baseline and bounded first step tracked in #3994
In-process HTTP dispatch adapter	SDK, Connect encoding, HTTP handlers and most existing enforcement	In-memory pipe and HTTP framing/transport machinery	Correct streaming, cancellation, headers/trailers, peer metadata and resource semantics; not a production use of a test response recorder
SDK-compatible local RPC binding	Typed SDK interfaces and shared protobuf schema	Wire serialization/framing for supported calls	Must explicitly reproduce the relevant validation, enforcement, ownership and RPC semantics
Narrow generated server-client facade	One typed schema and caller contract over local/remote adapters	Broad SDK dependency where unnecessary	An additive extension/migration decision, not a prerequisite for the initial pilot

Do not adopt direct service-to-service implementation calls that force callers to choose separate local and remote code paths. Shared protobuf structs alone do not guarantee behavioral parity.

The first design checkpoint must compare measured cost, reuse of existing enforcement, downstream compatibility, and maintenance burden. Select the smallest justified mechanism, not the one that removes the most layers on paper.

Why this is separate from the IPC v1 fix

Current IPC uses in-memory HTTP/2 pipes but still performs serialization, framing, allocations, and interceptor work. #3994 targets compression only and must remain independently mergeable. IPC v2 is a larger initiative; no speedup or zero-cost call claim is made in advance.

Proposed rollout contract

Illustrative startup configuration; exact names and the experimental binding identifier depend on the selected mechanism:

ipc:
  transport: connect-v1  # default
  # transport: experimental-v2

• Selection is startup-only, with explicit restart-based rollback to v1.
• Retain the current default. Validate supported service/client bindings at startup and report the active binding.
• This is a local deployment choice, not public capability negotiation for federated peers.
• Do not silently switch transports or retry through v1 after an ambiguous side-effecting v2 failure.
• A default change or v1 retirement needs separate approval, compatibility evidence, and a support window.
• The selector does not exist today; it is required before an experimental pilot.

IPC identity and authority boundary

At the inspected baseline, public RPCs install separate authn/authz interceptors; ordinary IPC uses a different interceptor that reconstructs trusted internal identity metadata without revalidating the token or running the general API authz interceptor. Selected routes (Rewrap by default, plus configured routes) perform token/DPoP reauthentication. Handler-owned policy checks are a separate concern. See server wiring and IPC handling.

Identity propagation is not an authorization grant. Preserve this existing IPC contract in the transport experiment; do not silently introduce public-path authz on every internal hop or remove existing checks. Explicit caller-authority, bounded service-authority, and delegation semantics are tracked separately in #4003. Shared middleware/KAS verification primitives are tracked in #4004; neither refactor is implicit in this transport experiment. Keep requester attribution distinct from the executing service and any entity being evaluated. Do not add an unscoped “already authorized” boolean or inherit arbitrary caller context as a shortcut.

Required parity contract

Define and test equivalent behavior for:

• Authentication/authorization and route-specific IPC reauthentication; locality is not authority.
• Verified identity, audit outcomes/causation, tracing, and relevant request metadata.
• Validation, response/status/error mapping, and panic handling.
• Deadlines, cancellation, shutdown/draining, concurrency, nested service calls, headers/trailers and streaming where supported.
• Request/response ownership and mutation isolation: a local adapter must not accidentally share mutable protobuf objects where serialization previously supplied a boundary.
• Meaningful message/resource limits and backpressure/admission behavior.

Preserve the approved security contract, not known attribution bugs. Do not bypass enforcement merely to improve benchmark numbers. Parity tests exercising writes must use isolated fixtures, not duplicate live production mutations.

Downstream SDK compatibility

Some downstream wrappers construct additional Connect clients from SDK.Conn().Client, .Endpoint, and .Options.

• A local HTTP adapter may be reusable by those wrappers if its supplied connection preserves their expected HTTP contract; test this rather than assume it.
• An upstream-only local RPC binding will not automatically accelerate clients that reconstruct the Connect path.
• A first pilot may intentionally mix upstream v2 and downstream v1 bindings, provided coverage/routing are visible and tested.
• Broader support may need an additive transport-neutral client factory. Preserve existing connection APIs during adoption.

Relevant source: SDK interfaces and construction, IPC construction, memhttp.

Staged delivery and acceptance

• Define the parity contract and reusable test/benchmark harness.
• Compare in-process HTTP dispatch and local RPC binding against IPC v1; document the chosen mechanism, rejected alternatives, and remaining gaps before committing to broad implementation.
• Prove one bounded existing service interface behind the new startup gate; no topology branches in service business logic.
• Test v1/v2/remote behavior, including mixed bindings, aliasing, cancellations, panics, failure mapping, shutdown, and limits. Declare unsupported streaming or custom-client cases explicitly.
• Benchmark representative workloads and separate transport costs from policy snapshots, caches, and downstream round trips.
• Publish the supported-binding inventory and reject unsupported required bindings at startup.
• Run an explicitly opted-in downstream reliability pilot, retaining v1 rollback and observability.
• Expand coverage only after parity/reliability gates pass and measurements justify it.
• Keep default changes or legacy removal out of the experimental implementation PRs.

Impact flags

Flag	Assessment
Requires downstream source changes	Conditional on dispatch mechanism and SDK wrapper usage; ordinary service business logic and request shapes must remain unchanged
Configuration/deployment coordination	Required for opt-in binding selection, reliability testing and restart rollback
Feature flag supported today	No — add the startup selector before the pilot
High complexity	Yes — changing a transport boundary requires semantic parity
High coupling	Yes — SDK bindings, enforcement, lifecycle and downstream extensions must agree

Branching and coordination

Keep #3994 independently mergeable. Prefer implementation branches from the same clean baseline, with orthogonal compression and transport-selection controls. A shared configuration namespace alone does not justify a prerequisite configuration PR. Introduce a common base only if implementation reveals a genuinely shared contract needed by both; otherwise reconcile the small config overlap during rebase.

Related: #3994 and #2520. Roadmap reference: ROAD-L04; baseline measurement: ROAD-S06.

V1 release classification

Parent: #3996. Class: post-v1. Score: 0/10 (correctness/reliability 0/3; compatibility 0/3; operability 0/2; explicit v1 dependency 0/2).

An alternative transport is an experiment, not a demonstrated platform release prerequisite. Existing SDK/Connect behavior remains supported while candidates are evaluated. These scores measure release necessity, not potential value. Reclassify only when evidence ties the work to an agreed v1 requirement. The experiment's own parity/rollout acceptance criteria do not, by themselves, make the experiment a v1 release gate.