Skip to content

agents: bound StatsD send queue processing - #632

Merged
santigimeno merged 1 commit into
node-v24.x-nsolid-v6.xfrom
santi/fix_statsd_agent
Oct 28, 2026
Merged

santigimeno merged 1 commit into
node-v24.x-nsolid-v6.xfrom
santi/fix_statsd_agent

Conversation

@santigimeno

Copy link
Copy Markdown
Member

Workers can refill the send queue faster than the StatsD thread drains it. Draining until empty can monopolize the agent loop and prevent endpoint reconfiguration and shutdown from running.

Process at most 1024 queued sends per callback and signal another pass when work remains. Keep the existing shutdown lock and FIFO ordering.

The worker reconfiguration stress test is unchanged. Native syntax compilation and whitespace checks pass; the runtime regression still needs a rebuilt binary.

Assisted-by: Codex

@ns-control-tower

Copy link
Copy Markdown

ns-control-tower

Note

Reviewing the latest changes on this PR — this may take a few minutes.
This comment will be updated in place with the findings; no other top-level comments will be posted.

@ns-control-tower

ns-control-tower commented Oct 26, 2026 •

Copy link
Copy Markdown

Walkthrough

agents/statsd/src/statsd_agent.cc: send_stats_msg_cb_ used to drain send_stats_msg_q_ until empty inside a single async callback. Under a worker flood — threads calling nsolid.statsd.sendRaw() / the native binding — one pass could run unbounded and monopolize the agent loop, delaying endpoint reconfiguration and shutdown. This change caps each pass at 1024 dequeues and, while the queue is still non-empty, re-arms send_stats_msg_ (uv_async_send) so the remainder is drained on a later loop turn. FIFO order and the stop_lock_ discipline are unchanged (+6/−1, one file).

This head (0d96c33c) is a force-push of the same commit: it wraps the previously over-long comment and was rebased onto the current base tip (node-v24.x-nsolid-v6.x @ 42202e45). The base move (otel changes landed by other PRs) is not part of this PR's diff. The PR was merged at 2026-10-28T10:14:05Z with this head.

overview
drain-flow-view

Assessment

  • ✔ Resolved — lint blocker: the comment at agents/statsd/src/statsd_agent.cc:769 is now wrapped across two lines (73/13 columns) and every changed line is ≤ 80 columns; the fresh Linters / lint-cpp run on this head passes (job). Local tools/cpplint.py + tools/checkimports.py on the file are also clean.
  • ✅ No lost wakeup: send() — the only producer — always enqueues then signals under the read lock (agents/statsd/src/statsd_agent.cc:779-786); the vendored libuv clears the async pending flag before invoking the callback, so any signal issued during or after a pass schedules another pass; a drained queue stops cleanly (no busy-loop; at most one coalescing-safe uv_async_send per pass).
  • ✅ Race-safe, order preserved: TSQueue::empty() and dequeue() are spinlock-guarded (src/nsolid/thread_safe.h:31-78), serialized with enqueue; a single drainer keeps FIFO order.
  • ✅ Shutdown path intact: the re-arm runs while holding stop_lock_ (read); do_stop() closes send_stats_msg_ only under the write lock (agents/statsd/src/statsd_agent.cc:564-578), so the new send() cannot reach a closing handle.
  • ✅ ASSERT_EQ (deps/asserts-cpp) is evaluated in release builds by design, so the re-signal is never compiled out; vendored uv_async_send returns 0 on all paths, so the assert cannot abort.
  • ℹ️ CI: on this head lint-cpp passed (it was the blocker on the previous head); the other Linters jobs passed (lint-js-and-md was still running — no JS/MD is changed), and the heavy runtime jobs (Test Linux/Test macOS/Coverage */Build from tarball) were still in progress. On the previous head their failures came from unrelated agent-test suites (gRPC profiling / ZMQ heap-profile), not StatsD.
  • (Re-review of force-pushed head 0d96c33c; the PR was merged before this run completed, so no separate review object is posted — this comment carries the verdict.)

Verdict: APPROVE — the only prior blocker (lint-cpp) is verifiably fixed on this head; the queue-bounding logic is unchanged from the full pass and remains correct.

@brycebaril brycebaril left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM though cpp-lint says this line is too long:

agents/statsd/src/statsd_agent.cc:769

Workers can refill the send queue faster than the StatsD thread drains
it. Draining until empty can monopolize the agent loop and prevent
endpoint reconfiguration and shutdown from running.

Process at most 1024 queued sends per callback and signal another pass
when work remains. Keep the existing shutdown lock and FIFO ordering.

The worker reconfiguration stress test is unchanged. Native syntax
compilation and whitespace checks pass; the runtime regression still
needs a rebuilt binary.

Fixes: #604
PR-URL: #632
Reviewed-By: Bryce Baril <bryce.baril@gmail.com>
Signed-off-by: Santiago Gimeno <santiago.gimeno@gmail.com>
@santigimeno
santigimeno force-pushed the santi/fix_statsd_agent branch from 34d7d7d to 0d96c33 Compare October 28, 2026 10:13
@santigimeno
santigimeno merged commit 0d96c33 into node-v24.x-nsolid-v6.x Oct 28, 2026
13 of 17 checks passed
@santigimeno
santigimeno deleted the santi/fix_statsd_agent branch October 28, 2026 10:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants