Skip to content

fix(otel): enforce unique roots and correlate X-Ray traces - #86

Merged
zhongkechen merged 2 commits into
mainfrom
codex/otel-unique-trace-roots
Aug 17, 2026
Merged

zhongkechen merged 2 commits into
mainfrom
codex/otel-unique-trace-roots

Conversation

@zhongkechen

@zhongkechen zhongkechen commented Aug 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This is the conformance companion to
aws-durable-execution-sdk-python#646.
It updates the OpenTelemetry contract and X-Ray validation path for three
related trace-integrity requirements:

  1. A trace ID must not contain multiple distinct root spans.
  2. An invocation-view continuation must not link to a previous operation span
    unless the SDK retained that span's real SpanContext.
  3. The X-Ray backend must retrieve both the ambient invocation trace and the
    separately rooted durable Workflow trace.

No new requirement IDs or SDK handlers are needed. The existing invocation,
execution, and long-running scenarios already exercise the affected trace
topologies.

Why

Multiple roots

The previous catalog asserted that Workflow was parentless, but it did not
reject another parentless span with the same trace ID. An SDK could therefore
emit a disconnected trace forest and still satisfy the individual span
assertions.

Multiple roots make backend behavior ambiguous:

  • a backend may choose the wrong span as the trace entry point or name;
  • trace duration, status, and root-cause attribution can be derived from the
    wrong branch;
  • critical-path and service-map relationships become disconnected;
  • root-based samplers can make independent decisions and retain only part of
    the logical trace.

Synthetic continuation links

The invocation catalog allowed or required links reconstructed from a
deterministic trace ID and span ID for a prior operation segment. Durable
checkpoints do not store that segment's actual SpanContext, so an SDK cannot
know whether the target span was created or exported, or reproduce its trace
flags and trace state. Such a link is synthetic rather than an observed causal
relationship.

Separately rooted X-Ray Workflow trace

The updated SDK keeps invocation spans in the ambient Lambda trace and gives the
durable Workflow span its own deterministic trace ID. ADOT translates the
parentless INTERNAL Workflow span into an X-Ray root segment named
Workflow, while the ambient trace remains indexed under the configured
service name.

The X-Ray validator previously queried only service("<configured service>").
It therefore retrieved invocation spans but could not see the separate
Workflow trace or its execution-view operation hierarchy.

Changes

Add require_unique_root_per_trace

The new provider-neutral assertion:

  • groups normalized spans by trace ID;
  • rejects more than one distinct parentless span in a group;
  • counts duplicate exports with the same trace ID and span ID once;
  • permits zero observed roots because some backends return a correlated subset
    without its upstream ambient root;
  • reports the names and span IDs of every conflicting root.

All 46 TelemetryAssertions and ExecutionTelemetryAssertions blocks in the
invocation, execution, and long-running suites enable the assertion.

Remove inferred previous-operation links

Invocation-view continuation and retry spans now require only their real link
to the durable Workflow span. This updates:

  • 9 continuation expectations in the standard invocation catalog;
  • 6 continuation alternatives in the long-running invocation catalog.

Execution-view links to actual Invocation spans are unchanged.

Correlate separate X-Ray traces

The X-Ray summary lookup now queries both:

service("<configured service>") OR service("Workflow")

The existing correlation logic then selects the Workflow trace as primary and
merges spans carrying the same durable execution ARN from ambient invocation
traces. A regression test covers two separate X-Ray trace IDs and verifies the
combined ordering.

Testing

  • 380 passed
  • focused X-Ray backend tests: 23 passed
  • mypy: 53 source files
  • repository-pinned Ruff formatting and lint
  • YAML validation
  • wheel and sdist metadata/resource checks
  • wheel-only extension discovery and OTel requirement smoke tests

@zhongkechen zhongkechen changed the title fix(otel): require unique roots per trace fix(otel): enforce unique roots and correlate X-Ray traces Aug 14, 2026
@zhongkechen
zhongkechen marked this pull request as ready for review August 14, 2026 23:21
@zhongkechen
zhongkechen temporarily deployed to ai-pr-review-runtime August 14, 2026 23:21 — with GitHub Actions Inactive
@zhongkechen
zhongkechen temporarily deployed to ai-pr-review-runtime August 14, 2026 23:21 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

Codex AI review

No actionable findings. Residual risk: live X-Ray service indexing is covered only by fake-client tests.

Reviewed commit 8dfb324aec1c4bda495ccd8e0381bded29f5b6bf. Workflow run

@github-actions

Copy link
Copy Markdown
Contributor

Claude AI review

This PR adds a provider-neutral require_unique_root_per_trace assertion, broadens the X-Ray summary filter to also match the separately-rooted Workflow service, and removes synthetic previous-operation links from the invocation and long-running catalogs.

The changes are correct and well-covered:

  • _unique_root_errors groups by each span's real trace_id, dedups duplicate exports by span_id, and permits 0–1 roots while rejecting 2+. It composes correctly with matching_trace, which preserves per-span trace_id, so the legitimate Workflow-root + ambient-root merge (distinct trace IDs) is not falsely flagged, and spans whose parent was filtered out of a correlated subset are not misread as roots. Boolean schema validation is added and ordered consistently with the updated validator test.
  • The X-Ray service(...) OR service("Workflow") filter only affects the summary-lookup path; unrelated Workflow traces are discarded by ARN correlation, so primary selection is unaffected. The new correlation test verifies two distinct X-Ray trace IDs merge into the expected ordering with the Workflow trace as primary.
  • Synthetic link removals are mirrored by the test_resources.py meta-tests and the catalog-coverage test, which enforces the new key on all assertion blocks.

No actionable correctness, typing, error-handling, or OpenTelemetry-compliance findings.

Residual test risk: the assertion is only unit-tested against the canonical model and X-Ray fakes. It bakes in the topology assumption that the durable Workflow span always carries a trace ID distinct from the ambient/operation spans. If any supported non-X-Ray backend (Datadog, Dash0, S3 collector) ever emits the Workflow root and another root under a single trace ID for a compliant SDK, enabling the key on all 46 blocks would surface as a conformance failure rather than a validator bug; end-to-end backend coverage is not exercised here.

Reviewed commit 8dfb324aec1c4bda495ccd8e0381bded29f5b6bf. Workflow run

@zhongkechen
zhongkechen merged commit 0dda42d into main Aug 17, 2026
17 of 48 checks passed
@zhongkechen
zhongkechen deleted the codex/otel-unique-trace-roots branch August 17, 2026 20:56

This branch was successfully deployed

1 inactive deployment
ai-pr-review-runtime — 8dfb324a Deployed Aug 14, 2026 by zhongkechen via ai-pr-review / Codex review / Generate Codex review #413
ai-pr-review — 8dfb324a Deployed Aug 14, 2026 by zhongkechen via ai-pr-review / Approve AI PR review #412
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants