[Feature]: Propagate per-operation trace context for chained invokes #952

What would you like?

Implement the SDK/plugin portion of per-operation X-Ray propagation so a chain-invoked Lambda execution can descend from the caller's CHAINED_INVOKE operation span. Support durable and non-durable targets; a non-durable target must not need the Durable Execution SDK merely to receive the tracing header.

This is a tracing-model and plugin-capability enhancement. The existing inherited-parent behavior remains the fallback when operation metadata is absent. The new capability lets an optional plugin contribute metadata to a persisted operation request; it must not change customer payloads or make durable execution depend on instrumentation.

Possible Implementation

Shared SDK/plugin contract

1. Add an optional synchronous propagation hook. The proposed contract is providePropagationMetadata(input) -> PropagationMetadata | absent, with idiomatic names in each SDK. The initial read-only input contains executionArn, operationId, parentOperationId, and targetFunctionName. Use SDK-owned input/result types; do not expose generated Lambda model objects, OTel types, or the mutable checkpoint request to plugins.
2. Collect results before the operation's START checkpoint is serialized. The hook is deterministic for the same execution/operation identity and side-effect free. Replaying an already checkpointed START must not invoke it again. A failed/uncommitted checkpoint can cause a later attempt to recompute metadata, so do not promise exactly-once hook invocation; recomputation must remain stable.
3. Merge supported metadata members in configured plugin order. For each member, the first non-null value wins. Different plugins may fill different members. If a later plugin supplies a different value for an already populated member, keep the first value, warn with both plugin identities, and increment a conflict counter. Omit empty metadata. Missing hooks, absent results and ordinary plugin failures must preserve the existing execution/fallback behavior; retain the SDK's fatal-error and cancellation rules.
4. Map the result to optional, flat wire fields in the core SDK: ChainedInvokeOptions.XAmznTraceId and DistributedMapOptions.XAmznTraceId. Metadata belongs to each operation update, because one checkpoint request can contain multiple invokes with different desired parents. Do not use a single tracing header on the checkpoint HTTP request as a substitute. Publish/consume the required service model and client dependency versions so serialization actually retains the new field.
5. Implement the producer in both OTel views. Use the resolved execution trace ID, the actual deterministic span ID for the calling operation, and the resolved sampling decision. Emit X-Ray header encoding, for example Root=1-<8 hex>-<24 hex>;Parent=<operation span ID>;Sampled=0|1. The plugin owns tracing/encoding; the core transports the resulting string. Do not send a W3C 00-... value in this field, adopt a transient checkpoint HTTP span as the operation parent, or change cross-language ID algorithms as part of this work.
6. Keep the contract extensible without adding unrelated operation APIs. The model proposal also includes distributed-map propagation and names fanout/HTTP START paths. Preserve the optional field in applicable models/serializers, and confirm the operation-specific inputs and supported dispatch paths before wiring those paths. New operation APIs or end-to-end W3C/baggage propagation need their own scope. Customer payload and ClientContext propagation remain customer-owned.

Repository-specific work

• Add the optional, synchronous return-value hook and read-only SDK types to the plugin API. Implement result aggregation/error isolation in the composite dispatcher; the existing Promise-returning observer hooks are not a replacement for collecting metadata before serialization.
• In the invoke handler, obtain metadata after stable operation identity/target information exists and before building the new START update. The current onOperationStart observer is called after the checkpoint and also on replay, so it cannot provide this behavior unchanged.
• Map the optional header through the Lambda client command input/serializer, including supported distributed-map options, and update @aws-sdk/client-lambda when the generated model is available.
• Add the producer to execution/invocation OTel providers using the existing canonical trace and operation span-ID helpers. Cover the public wrapper, local-testing path and the supported registration paths. Coordinate with the per-invocation plugin work in PR refactor(sdk): create one plugin instance per invocation #924.

Acceptance criteria

• Existing plugins with no propagation hook and executions without an OTel plugin keep working with the new fields omitted.
• A new invoke START carries the expected operation-specific header; no propagation callback is run when replay consumes an existing checkpoint. Cover failed checkpoint/retry recomputation without asserting exactly-once hook delivery.
• Two invokes batched in one checkpoint carry distinct operation parent IDs; tenant, target and payload fields are preserved.
• Merge tests cover absent values, different members, equal repeated values, conflicting values, ordering, warning/counter behavior, and throwing plugins without changing the execution outcome.
• Both OTel views preserve canonical trace identity and sampled/unsampled decisions, with inherited context and supported fallback-context cases.
• Serialization/round-trip tests prove the generated/custom client models retain optional XAmznTraceId fields, including distributed-map models where supported; older inputs without the fields still work.
• Once backend support is available, deployed tests cover durable and non-durable children, mixed-language parent/child combinations, suspension/resume and both Active and PassThrough tracing. Under Active tracing, Lambda may insert its own segment: assert the correct ancestry rather than requiring every child Workflow to have the operation as its immediate parent.
• Coordinate updated topology requirements and language handlers with the conformance repository. Scope the new assertions to operation-level propagation while retaining the old fallback contract.
• Document the hook, ordering/failure/replay semantics, supported versions, and rollout dependencies before declaring the end-to-end capability complete.

Is this a breaking change?

The new wire members and hook should be optional and additive; existing plugin implementations should not be forced to implement a new method. Enabling operation-level propagation intentionally changes the child trace hierarchy, so the telemetry contract, affected tests and release/migration guidance need review. Coordinate with any ongoing plugin-factory API migration rather than introducing avoidable successive registration changes.

Does this require an RFC?

Yes: this adds a plugin capability that contributes to persisted operation requests and changes trace topology. Finalize the SDK-owned metadata type/member names and generic input shape during shared design review. Keep the reviewed flat XAmznTraceId wire shape and X-Ray encoding consistent across implementations; older nested/W3C examples are not a separate wire contract.

Additional Context

Current source inspected at 2d285ddb54d07be3dcbdc09f7fd1696aece67659. Proposed language-specific hook name: providePropagationMetadata.

• Invoke START construction
• Plugin contracts
• Composite plugin dispatch
• Execution-view OTel producer
• Invocation-view OTel producer

The LMI inbound-header issue #951 is separate: this request concerns outbound context on persisted invoke operations. The completed tracing work in #854 does not supply this per-operation checkpoint field.

Service-model publication and backend forwarding/rollout are external dependencies. This issue tracks the SDK and OTel-plugin implementation, not backend deployment. The backend must accept the operation-level header and retain its normalized result for child invocations; SDK-only changes cannot establish the new hierarchy by themselves. No new end-to-end implementation or runtime validation is claimed by this tracking issue.

Related SDK implementation tracking: Java #764, Python #751.