You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(otel): enforce unique roots and correlate X-Ray traces - #86
This is the conformance companion to aws-durable-execution-sdk-python#646.
It updates the OpenTelemetry contract and X-Ray validation path for three
related trace-integrity requirements:
A trace ID must not contain multiple distinct root spans.
An invocation-view continuation must not link to a previous operation span
unless the SDK retained that span's real SpanContext.
The X-Ray backend must retrieve both the ambient invocation trace and the
separately rooted durable Workflow trace.
No new requirement IDs or SDK handlers are needed. The existing invocation,
execution, and long-running scenarios already exercise the affected trace
topologies.
Why
Multiple roots
The previous catalog asserted that Workflow was parentless, but it did not
reject another parentless span with the same trace ID. An SDK could therefore
emit a disconnected trace forest and still satisfy the individual span
assertions.
Multiple roots make backend behavior ambiguous:
a backend may choose the wrong span as the trace entry point or name;
trace duration, status, and root-cause attribution can be derived from the
wrong branch;
critical-path and service-map relationships become disconnected;
root-based samplers can make independent decisions and retain only part of
the logical trace.
Synthetic continuation links
The invocation catalog allowed or required links reconstructed from a
deterministic trace ID and span ID for a prior operation segment. Durable
checkpoints do not store that segment's actual SpanContext, so an SDK cannot
know whether the target span was created or exported, or reproduce its trace
flags and trace state. Such a link is synthetic rather than an observed causal
relationship.
Separately rooted X-Ray Workflow trace
The updated SDK keeps invocation spans in the ambient Lambda trace and gives the
durable Workflow span its own deterministic trace ID. ADOT translates the
parentless INTERNAL Workflow span into an X-Ray root segment named Workflow, while the ambient trace remains indexed under the configured
service name.
The X-Ray validator previously queried only service("<configured service>").
It therefore retrieved invocation spans but could not see the separate
Workflow trace or its execution-view operation hierarchy.
Changes
Add require_unique_root_per_trace
The new provider-neutral assertion:
groups normalized spans by trace ID;
rejects more than one distinct parentless span in a group;
counts duplicate exports with the same trace ID and span ID once;
permits zero observed roots because some backends return a correlated subset
without its upstream ambient root;
reports the names and span IDs of every conflicting root.
All 46 TelemetryAssertions and ExecutionTelemetryAssertions blocks in the
invocation, execution, and long-running suites enable the assertion.
Remove inferred previous-operation links
Invocation-view continuation and retry spans now require only their real link
to the durable Workflow span. This updates:
9 continuation expectations in the standard invocation catalog;
6 continuation alternatives in the long-running invocation catalog.
Execution-view links to actual Invocation spans are unchanged.
Correlate separate X-Ray traces
The X-Ray summary lookup now queries both:
service("<configured service>") OR service("Workflow")
The existing correlation logic then selects the Workflow trace as primary and
merges spans carrying the same durable execution ARN from ambient invocation
traces. A regression test covers two separate X-Ray trace IDs and verifies the
combined ordering.
Testing
380 passed
focused X-Ray backend tests: 23 passed
mypy: 53 source files
repository-pinned Ruff formatting and lint
YAML validation
wheel and sdist metadata/resource checks
wheel-only extension discovery and OTel requirement smoke tests
This PR adds a provider-neutral require_unique_root_per_trace assertion, broadens the X-Ray summary filter to also match the separately-rooted Workflow service, and removes synthetic previous-operation links from the invocation and long-running catalogs.
The changes are correct and well-covered:
_unique_root_errors groups by each span's real trace_id, dedups duplicate exports by span_id, and permits 0–1 roots while rejecting 2+. It composes correctly with matching_trace, which preserves per-span trace_id, so the legitimate Workflow-root + ambient-root merge (distinct trace IDs) is not falsely flagged, and spans whose parent was filtered out of a correlated subset are not misread as roots. Boolean schema validation is added and ordered consistently with the updated validator test.
The X-Ray service(...) OR service("Workflow") filter only affects the summary-lookup path; unrelated Workflow traces are discarded by ARN correlation, so primary selection is unaffected. The new correlation test verifies two distinct X-Ray trace IDs merge into the expected ordering with the Workflow trace as primary.
Synthetic link removals are mirrored by the test_resources.py meta-tests and the catalog-coverage test, which enforces the new key on all assertion blocks.
No actionable correctness, typing, error-handling, or OpenTelemetry-compliance findings.
Residual test risk: the assertion is only unit-tested against the canonical model and X-Ray fakes. It bakes in the topology assumption that the durable Workflow span always carries a trace ID distinct from the ambient/operation spans. If any supported non-X-Ray backend (Datadog, Dash0, S3 collector) ever emits the Workflow root and another root under a single trace ID for a compliant SDK, enabling the key on all 46 blocks would surface as a conformance failure rather than a validator bug; end-to-end backend coverage is not exercised here.
Reviewed commit 8dfb324aec1c4bda495ccd8e0381bded29f5b6bf. Workflow run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is the conformance companion to
aws-durable-execution-sdk-python#646.It updates the OpenTelemetry contract and X-Ray validation path for three
related trace-integrity requirements:
unless the SDK retained that span's real
SpanContext.separately rooted durable
Workflowtrace.No new requirement IDs or SDK handlers are needed. The existing invocation,
execution, and long-running scenarios already exercise the affected trace
topologies.
Why
Multiple roots
The previous catalog asserted that
Workflowwas parentless, but it did notreject another parentless span with the same trace ID. An SDK could therefore
emit a disconnected trace forest and still satisfy the individual span
assertions.
Multiple roots make backend behavior ambiguous:
wrong branch;
the logical trace.
Synthetic continuation links
The invocation catalog allowed or required links reconstructed from a
deterministic trace ID and span ID for a prior operation segment. Durable
checkpoints do not store that segment's actual
SpanContext, so an SDK cannotknow whether the target span was created or exported, or reproduce its trace
flags and trace state. Such a link is synthetic rather than an observed causal
relationship.
Separately rooted X-Ray Workflow trace
The updated SDK keeps invocation spans in the ambient Lambda trace and gives the
durable
Workflowspan its own deterministic trace ID. ADOT translates theparentless
INTERNALWorkflow span into an X-Ray root segment namedWorkflow, while the ambient trace remains indexed under the configuredservice name.
The X-Ray validator previously queried only
service("<configured service>").It therefore retrieved invocation spans but could not see the separate
Workflow trace or its execution-view operation hierarchy.
Changes
Add
require_unique_root_per_traceThe new provider-neutral assertion:
without its upstream ambient root;
All 46
TelemetryAssertionsandExecutionTelemetryAssertionsblocks in theinvocation, execution, and long-running suites enable the assertion.
Remove inferred previous-operation links
Invocation-view continuation and retry spans now require only their real link
to the durable
Workflowspan. This updates:Execution-view links to actual
Invocationspans are unchanged.Correlate separate X-Ray traces
The X-Ray summary lookup now queries both:
The existing correlation logic then selects the Workflow trace as primary and
merges spans carrying the same durable execution ARN from ambient invocation
traces. A regression test covers two separate X-Ray trace IDs and verifies the
combined ordering.
Testing
380 passed23 passed53 source files