feat(orchestrator): Telemetry - Trace the execution output artifacts info retrieval time - #335
Conversation
This stack of pull requests is managed by Graphite. Learn more about stacking. |
| ) | ||
| for output_name, uri in output_artifact_uris.items() | ||
| } | ||
| span.set_attribute( |
There was a problem hiding this comment.
I'm not sure this is useful. There are many executions that output huge multi-gig artifacts. And they process super-fast.
Reporting the number of files would have been OK, but we don't have that at this moment.
P.S. If span reporting cannot work without attributes, then keep it of course.
There was a problem hiding this comment.
I can remove this span attribute. Thank you for letting me know.,
There was a problem hiding this comment.
Done — removed the output.total_size attribute and pushed. The span is now timing-only, which is exactly the signal we want here (the get_info() dir-walk duration). Agreed on file count being the meaningful dimension; we don't have it cheaply today, so leaving it out rather than reporting a misleading size.
Ark-kun
changed the title
Ark-kun
changed the title
Ark-kun
changed the title
70cd297 to
752c063
Compare
Merge activity
|
752c063 to
9f43f9d
Compare
2 participants
Wraps the output-artifact
get_info()resolution in theSUCCEEDEDbranch ofinternal_process_one_running_executionin a child span (orchestrator.get_output_artifact_info).Why
This
get_reader().get_info()call walks and stats every file of each output artifact on the single orchestrator thread. For a large Directory artifact (the inv-26772 trigger: ~88k per-merchant files) this is the confirmed freeze hot path — it runs for 40+ minutes and blocks all other executions. The span isolates that hot path within the parent processing span so its duration is directly measurable.Timing-only, by design: per review, artifact byte size does not predict the stall (multi-gig artifacts resolve fast); file count is the meaningful dimension but is not available cheaply at this point, so no attribute is attached rather than a misleading one.
Impact
One child span per succeeded execution's output resolution, nested under
orchestrator.process_running_executionfrom #334. No behavior change.