Skip to content

cli: off-CPU flame, time windows, run markers, and Linux QOL - #7

Merged
fasterthanlime merged 1 commit into
mainfrom
cli-offcpu-windows-markers
Aug 6, 2026
Merged

fasterthanlime merged 1 commit into
mainfrom
cli-offcpu-windows-markers

Conversation

@fasterthanlime

Copy link
Copy Markdown
Collaborator

Stall-forensics features for agents/humans plus QOL fixes for papercuts hit while profiling a root-owned render service (binnacle render, a Blitz/Vello panel) on Linux. All changes verified live against that target.

Features

stax flame --off-cpu — drive the flame from the off-CPU breakdown the server already aggregates per FlameNode. This is the view for "what was this thread stuck on during the freeze?": a parked thread yields ~no on-CPU samples, so an on-CPU flame cannot show it. Each node prints its dominant blocking reason (lock/sleep/io/ready/…).

$ stax flame --off-cpu --tid 616281
 off-CPU  reason     %  frame
 5422.35   ready 100.0  (root)
 4573.11   ready  84.3    └─ <vox_types::channel::Rx<Snapshot>>::recv  → do_epoll_wait

--window on flame/top — a small duration language resolved into the existing LiveFilter.time_range (which the server already honours; aggregate_with_filter handles it). Forms: 30s (trailing), 1m..90s (absolute), A.., ..B. Duration anchors to the perf clock (TimelineUpdate.recording_duration_ns), not wall-clock, so windows align with the data — the wall-clock path would silently misalign by the boot offset.

stax mark <label> + --window <marker>.. — drop a named marker into the active run, then anchor a query at it:

$ stax mark smoke-start
$ stax mark click-done
$ stax flame --tid 616278 --window smoke-start..click-done

Markers persist in the saved aggregator (archives keep their anchors) and surface on the timeline. Stamped with last_event_ns (the recording's own clock) so no wall-clock→perf conversion is needed.

QOL fixes (each hit for real today)

  • EACCES vs absent socket: require_server_socket used .exists(), which returns false for a root-owned 0600 socket the non-root CLI can't stat → reported "isn't running". Now probes access(W_OK|R_OK) and reports the cause with concrete escapes (sudo, or the ws:// endpoint).
  • ws:// transport for the CLI: STAX_SERVER_SOCKET=ws://host:port now reaches a server without filesystem access to its unix socket. Enables vox transport-websocket in the CLI (it was already on for the server).
  • Recorder mlock EPERM: the perf-ring mmap failure now names kernel.perf_event_mlock_kb and the sysctl to raise it, instead of bare "Operation not permitted" (the default ~516KB budget is smaller than one ring).
  • --dwarf-unwind docs: AGENTS.md referenced a nonexistent positive flag; the real flag is --no-dwarf-unwind (unwinding is on by default). Fixed the three references.
  • WS bind collision names the address and the STAX_SERVER_WS_BIND escape instead of bare "Address already in use".
  • sudo stax setup --server-root: installs a root stax-server systemd service for profiling root-owned targets — the per-user server can't read their /proc/<pid>/maps, so every symbol resolves to <unresolved>. This is the one-command answer to today's multi-hour fight.

Verification

  • cargo test --workspace --lib --tests green (82 tests; the failing stax-linux-capture examples are pre-existing upstream drift — they use struct-literal RecordOptions missing the dwarf_unwind field upstream already added; verified failing on clean HEAD, unrelated to this PR).
  • Each feature exercised live against a running binnacle render (pid 616278): off-CPU flame shows the broker-link pump parked in Snapshot::recv → epoll_wait with reason ready; marker windows slice the click workload cleanly (idle baseline 0.47s active vs post-click 5.35s).

Add three agent-facing stall-forensics features and fix six QOL
papercuts hit while profiling a root-owned render service.

Features:
- `stax flame --off-cpu`: drive the flame from the off-CPU breakdown
  the server already aggregates per node. The view for "what was this
  thread stuck on during the freeze?" — a parked thread yields ~no
  on-CPU samples, so an on-CPU flame cannot show it.
- `--window` on flame/top: a small duration language (`30s`,
  `1m..90s`, `A..`, `..B`) resolved into the existing
  `LiveFilter.time_range`, which the server already honours.
  Duration comes from the perf clock (timeline `recording_duration_ns`),
  not wall-clock, so windows align with data.
- `stax mark <label>`: drop a named marker into the active run, then
  `--window freeze..` anchors a query at that point. Markers persist in
  the saved aggregator and surface on the timeline.

QOL fixes:
- Distinguish a root-owned / unreadable server socket from a missing
  one; report EACCES with a concrete escape instead of "isn't running".
- Let `STAX_SERVER_SOCKET` take a `ws://host:port` URL so a root or
  remote server can be reached without filesystem access to its socket
  (enables vox `transport-websocket` in the CLI).
- Recorder EPERM on perf ring mmap names `kernel.perf_event_mlock_kb`
  and the sysctl to raise it, instead of a bare "Operation not permitted".
- Fix AGENTS.md references to a nonexistent `--dwarf-unwind` flag; the
  real flag is `--no-dwarf-unwind` (unwinding is on by default).
- WS bind collision names the address and the `STAX_SERVER_WS_BIND`
  escape instead of bare "Address already in use".
- `sudo stax setup --server-root` installs a root stax-server systemd
  service for profiling root-owned targets (the user server cannot read
  their /proc/<pid>/maps, so every symbol resolves to <unresolved>).
@fasterthanlime
fasterthanlime merged commit 83e3b8f into main Aug 6, 2026
1 of 4 checks passed
@fasterthanlime
fasterthanlime deleted the cli-offcpu-windows-markers branch August 6, 2026 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant