Repository navigation
cli: off-CPU flame, time windows, run markers, and Linux QOL - #7
Merged
Merged
Conversation
Add three agent-facing stall-forensics features and fix six QOL papercuts hit while profiling a root-owned render service. Features: - `stax flame --off-cpu`: drive the flame from the off-CPU breakdown the server already aggregates per node. The view for "what was this thread stuck on during the freeze?" — a parked thread yields ~no on-CPU samples, so an on-CPU flame cannot show it. - `--window` on flame/top: a small duration language (`30s`, `1m..90s`, `A..`, `..B`) resolved into the existing `LiveFilter.time_range`, which the server already honours. Duration comes from the perf clock (timeline `recording_duration_ns`), not wall-clock, so windows align with data. - `stax mark <label>`: drop a named marker into the active run, then `--window freeze..` anchors a query at that point. Markers persist in the saved aggregator and surface on the timeline. QOL fixes: - Distinguish a root-owned / unreadable server socket from a missing one; report EACCES with a concrete escape instead of "isn't running". - Let `STAX_SERVER_SOCKET` take a `ws://host:port` URL so a root or remote server can be reached without filesystem access to its socket (enables vox `transport-websocket` in the CLI). - Recorder EPERM on perf ring mmap names `kernel.perf_event_mlock_kb` and the sysctl to raise it, instead of a bare "Operation not permitted". - Fix AGENTS.md references to a nonexistent `--dwarf-unwind` flag; the real flag is `--no-dwarf-unwind` (unwinding is on by default). - WS bind collision names the address and the `STAX_SERVER_WS_BIND` escape instead of bare "Address already in use". - `sudo stax setup --server-root` installs a root stax-server systemd service for profiling root-owned targets (the user server cannot read their /proc/<pid>/maps, so every symbol resolves to <unresolved>).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stall-forensics features for agents/humans plus QOL fixes for papercuts hit while profiling a root-owned render service (
binnacle render, a Blitz/Vello panel) on Linux. All changes verified live against that target.Features
stax flame --off-cpu— drive the flame from the off-CPU breakdown the server already aggregates perFlameNode. This is the view for "what was this thread stuck on during the freeze?": a parked thread yields ~no on-CPU samples, so an on-CPU flame cannot show it. Each node prints its dominant blocking reason (lock/sleep/io/ready/…).--windowon flame/top — a small duration language resolved into the existingLiveFilter.time_range(which the server already honours;aggregate_with_filterhandles it). Forms:30s(trailing),1m..90s(absolute),A..,..B. Duration anchors to the perf clock (TimelineUpdate.recording_duration_ns), not wall-clock, so windows align with the data — the wall-clock path would silently misalign by the boot offset.stax mark <label>+--window <marker>..— drop a named marker into the active run, then anchor a query at it:Markers persist in the saved aggregator (archives keep their anchors) and surface on the timeline. Stamped with
last_event_ns(the recording's own clock) so no wall-clock→perf conversion is needed.QOL fixes (each hit for real today)
require_server_socketused.exists(), which returns false for a root-owned 0600 socket the non-root CLI can't stat → reported "isn't running". Now probesaccess(W_OK|R_OK)and reports the cause with concrete escapes (sudo, or the ws:// endpoint).STAX_SERVER_SOCKET=ws://host:portnow reaches a server without filesystem access to its unix socket. Enables voxtransport-websocketin the CLI (it was already on for the server).kernel.perf_event_mlock_kband the sysctl to raise it, instead of bare "Operation not permitted" (the default ~516KB budget is smaller than one ring).--dwarf-unwinddocs: AGENTS.md referenced a nonexistent positive flag; the real flag is--no-dwarf-unwind(unwinding is on by default). Fixed the three references.STAX_SERVER_WS_BINDescape instead of bare "Address already in use".sudo stax setup --server-root: installs a rootstax-serversystemd service for profiling root-owned targets — the per-user server can't read their/proc/<pid>/maps, so every symbol resolves to<unresolved>. This is the one-command answer to today's multi-hour fight.Verification
cargo test --workspace --lib --testsgreen (82 tests; the failingstax-linux-captureexamples are pre-existing upstream drift — they use struct-literalRecordOptionsmissing thedwarf_unwindfield upstream already added; verified failing on clean HEAD, unrelated to this PR).binnacle render(pid 616278): off-CPU flame shows the broker-link pump parked inSnapshot::recv → epoll_waitwith reasonready; marker windows slice the click workload cleanly (idle baseline 0.47s active vs post-click 5.35s).