Repository navigation
agents: export span events and links - #609
Conversation
WalkthroughExports span events, recorded exceptions and Re-review of the delta since the last review ( Assessment
Verdict: APPROVE — the only blocker from the previous review (the |
d58a40d to
c2936b0
Compare
brycebaril
left a comment
There was a problem hiding this comment.
Thanks, Santi. Checked at c2936b0. One attribute converter for spans, events and links is the right shape. The link checks in fill_recordable hold up: hex length and digits, flags from 0 to 255, and a valid context. The test covers all three exporters, bad inputs included.
Two things before merge:
- Nothing caps events or links any more. Until now
s.eventsnever left the process, so their number didn't matter. Now every event and link goes out, and the exporters' options leavemax_events,max_linksand the per-event and per-link attribute limits atuint32max:agents/grpc/src/grpc_agent.cc:1124, andsetup_trace_otlp_exporterandsetup_trace_grpc_otlp_exporterinagents/otlp/src/otlp_agent.cc. A long-lived span that adds an event per item or per retry can carry thousands. One export over the receiver's message limit (4 MB by default in gRPC) fails with every span in it. The spec's defaults, 128 each, on those three sets of options letOtlpRecordabledrop the rest and reportdropped_events_countanddropped_links_count. - An event time from
performance.now()lands in 1970. The OTel API'sTimeInputis "hrtime, epoch milliseconds, performance.now() or Date", and the JS SDK reads a number below the time origin asperformance.now().fill_recordabletakes every number as epoch milliseconds (otlp_common.cc:744), sospan.addEvent('x', {}, performance.now())is exported a few seconds after 1970, andrecordExceptionwith such a time too. InaddEvent, addinggetTimeOriginTimestamp()whenstartTimeis a number below it fixes both; the helper is already imported.
CI's test jobs on c2936b0 ended in a runner shutdown (exit 143) while still building, so the new test hasn't run yet; a re-run settles it. The Windows build fails in libsodium on other PRs on this base too.
Optional notes are inline.
| } else { | ||
| continue; | ||
| } | ||
| recordable->AddEvent(name.get<std::string>(), |
There was a problem hiding this comment.
Before merge: from here on nothing limits how many events a span sends. The exporters' options leave max_events and max_links, and the per-event and per-link attribute limits, at uint32 max. So a span with thousands of events goes out whole, and an export over the receiver's limit fails with all its spans. The spec's defaults, 128 each, on the options at grpc_agent.cc:1124 and in the two setup_trace_* functions in otlp_agent.cc let OtlpRecordable drop the rest and count them.
| uint64_t timestamp; | ||
| const auto max_timestamp = std::numeric_limits<int64_t>::max(); | ||
| if (time.is_number()) { | ||
| const double nanos = time.get<double>() * 1e6; |
There was a problem hiding this comment.
Before merge: every number is taken as epoch milliseconds here, but TimeInput also allows performance.now(), which the JS SDK tells apart as a number below the time origin. Such an event is exported a few seconds after 1970. Adding getTimeOriginTimestamp() in addEvent when startTime is a number below it covers recordException too, since it goes through addEvent.
|
|
||
| const api = getApi(); | ||
| for (const link of links) { | ||
| if (!api.trace.isSpanContextValid(link.context)) continue; |
There was a problem hiding this comment.
Optional: the test shows a bad link is meant to be skipped, but two kinds throw out of startSpan instead. isSpanContextValid reads link.context.traceId, so a link with an undefined context throws; trace.getSpanContext() returns undefined for a context with no span. Line 85 calls serialize() on any traceState, so a plain string throws too. Checking link?.context first, and typeof context.traceState?.serialize === 'function' on line 85, skips both like the other bad links.
| attributes = undefined; | ||
| } | ||
|
|
||
| if (typeof startTime === 'object' && startTime !== null && |
There was a problem hiding this comment.
Optional: DatePrototypeGetTime throws for any object that isn't a Date, so span.addEvent('x', {}, {}) throws where other bad times are dropped. startTime instanceof Date, as isTimeInput checks, keeps this to Dates.
c2936b0 to
f1bbb1e
Compare
brycebaril
left a comment
There was a problem hiding this comment.
Thanks, Santi. Checked at c10450c. The rebase left the first commit as I reviewed it, and the four commits after it cover everything from my last review:
- Caps. All three trace exporters now set
max_events,max_linksand the per-event and per-link attribute limits to the SDK's 128:grpc_agent.cc:1126-1131,setup_trace_otlp_exporterandsetup_trace_grpc_otlp_exporter. Both agents get their recordables fromMakeRecordable(), which passes those options toOtlpRecordable, so it keeps 128 of each and counts the rest as dropped. Span attributes stay unlimited, as before. performance.now()times.addEventadds the time origin to a number between 0 and the origin, as the JS SDK does, andrecordExceptiongoes through it.- Malformed links. A link with no context, or with a trace state that has no
serialize, is skipped instead of throwing out ofstartSpan. One whoseserializereturns something other than a string is dropped on the native side. - Object times. Only Dates are converted, so
{}and an invalid Date are dropped instead of throwing.
The test checks all of this on all three exporters, and its checks fail without the fixes:
- 128 events and links with one dropped, and 128 attributes on each with one dropped.
performance.now()times that land between the span's start and now.- The malformed links and times.
It passed on macOS and in the Linux coverage job. The other failures on this head are the ones every PR on this base has: Linux runners shut down mid-run, the Windows build fails in libsodium, and the macOS failures are tests that fail on other PRs too.
Approving. One thing for a separate issue, not this PR: span start and end times still accept only performance.now() values.
startSpan(name, { startTime })andspan.end(endTime)send the value as given, andSpan::add_propadds the process start time to it (src/nsolid/nsolid_trace.cc:47-70).- So a start given in epoch milliseconds lands decades ahead.
- A Date or an HrTime array reaches
pushSpanDataDouble, which expects a number.
Converting them in JS, as addEvent now does, would match the JS SDK.
Include events and recorded exceptions in gRPC and OTLP spans, and pass startSpan() links through native storage to the shared exporter. Reuse attribute conversion for spans, events and links, validate payload types and normalize event timestamps, preserving hrtime nanoseconds. Apply the SDK defaults of 128 events, links and attributes per event or link, with dropped counts. Add regression coverage for all three export paths, malformed inputs, timestamps and limits. Fixes: #569 PR-URL: #609 Reviewed-By: Bryce Baril <bryce.baril@gmail.com> Signed-off-by: Santiago Gimeno <santiago.gimeno@gmail.com>
c10450c to
dad7f7c
Compare
Include events and recorded exceptions in gRPC and OTLP spans. Pass links from startSpan() through native storage to the shared exporter, including their attributes and trace context.
Reuse attribute conversion for spans, events and links, and preserve event timestamps, including hrtime nanoseconds.
Fixes: #569