[dbshah12]

Background

PR #128 (DLPX-98742) rewrote connstat-stats.sh to use a single long-running connstat process piped through stdbuf -oL awk, replacing the previous connstat -c 2 loop. The goal was to eliminate the 20 s batch-delivery latency and produce a continuous per-line stream to Telegraf's execd input.

The PR comment stated that stdbuf -oL forces line-buffered stdout on awk. That claim is correct for gawk but insufficient for mawk, the default awk on every stock Ubuntu 24.04 engine — because the real problem is on the input side, not the output side.

Problem

mawk 1.3.4 block-reads its stdin: it keeps calling read() to fill a 256 KB input buffer before processing any record. stdbuf -oL does take effect on mawk's stdout (verified with strace below); it just can't help in this pipeline, because mawk never gets as far as writing. fflush() has the same limitation.

connstat writes only a few KB per 10 s interval and never exits (-c removed in PR #128), so mawk's input buffer never fills, EOF never arrives, and Telegraf's execd input receives zero tcp_stats data — indefinitely, with no error logged on either side.

strace evidence (credit: @prakashsurya during code review):

# 1. Default mawk, fflush() after every print: reads keep filling the 256 KB buffer, nothing is written
$ (while :; do echo a,b; sleep 1; done) | strace -e trace=read,write mawk -F, '{print; fflush()}' >/dev/null
read(0, "a,b\n", 262144)                = 4
read(0, "a,b\n", 262140)                = 4
read(0, "a,b\n", 262136)                = 4

# 2. -W interactive: each line is read, then written straight away
$ (while :; do echo a,b; sleep 1; done) | strace -e trace=read,write mawk -W interactive -F, '{print}' >/dev/null
read(0, "a,b\n", 4096)                  = 4
write(1, "a,b", 3)                      = 3
write(1, "\n", 1)                       = 1

# 3. stdbuf -oL DOES affect mawk's stdout (one write per line, versus a single batched write without stdbuf):
$ strace -f -e trace=write stdbuf -oL mawk 'BEGIN{print "a"; print "b"}' >/dev/null
write(1, "a\n", 2)                      = 2
write(1, "b\n", 2)                      = 2
$ strace -e trace=write mawk 'BEGIN{print "a"; print "b"}' >/dev/null
write(1, "a\nb\n", 4)                   = 4

Case #3 rules out "mawk bypasses libc stdio" — stdbuf can reach mawk's stdout, it just doesn't matter when mawk is stuck in a read() loop filling an input buffer that never fills.

Why PR #128 Appeared to Work

PR #128 was validated exclusively on dhruv5.dlpxdc.co (2026.5.0.0), which had gawk installed as the awk alternative (/usr/bin/awk → /usr/bin/gawk). gawk reads stdin line by line and emits each record immediately — the pipeline streams correctly regardless of stdbuf.

On a stock Ubuntu 24.04 engine, mawk is the only awk installed (Priority: required) and is the default alternative (/usr/bin/awk → /usr/bin/mawk). On those engines, PR #128 silently broke tcp_stats collection entirely from the moment it landed.

Fix

Replace stdbuf -oL awk with /usr/bin/mawk -W interactive.

-W interactive is mawk's native flag that switches stdin to line-buffered reads (and also unbuffers stdout). It works regardless of libc or LD_PRELOAD.

-    | stdbuf -oL awk -F',' '
+    | /usr/bin/mawk -W interactive -F',' '

Additional changes in commit 4b14f5a (in response to code review):

• Script comment rewritten to describe the input-side cause.
• Dropped the two fflush() calls — redundant under -W interactive.
• Added mawk to debian/control Depends: — mawk is Priority: required on Ubuntu 24.04 so already present, but this records the hard runtime dependency explicitly.

Test Plan

Root cause reproduction

Tested on two engines: one where the bug was masked and one where it was visible.

dlpx-dev.dlpxdc.co — awk → gawk (non-standard, same configuration as dhruv5 in PR #128):

Command	Lines in 15 s
connstat … | stdbuf -oL mawk …	0 — mawk's 256 KB input buffer never fills
connstat … | mawk -W interactive …	3 — line-buffered reads, data flows immediately
connstat … | stdbuf -oL gawk …	3 — gawk reads line by line regardless of stdbuf

dlpx-qa-develop-qar-233997-766f97e2.dlpxdc.co (Azure) — awk → mawk (standard configuration):

Command	Lines in 15 s
connstat … | stdbuf -oL awk … (current script)	0 — bug confirmed
connstat … | mawk -W interactive … (fix)	18 — data flows immediately

Live InfluxDB verification

Deployed the final script (post commit 4b14f5a) to both engines, restarted Telegraf, waited 5 min, queried the default bucket for tcp_stats:

Engine	awk alternative	tcp_stats points (last 5 min)	Series	Fields
dlpx-dev.dlpxdc.co	gawk	660	3 (https, ssh, ldap)	all 10 ✓
dlpx-qa-develop-qar-233997-766f97e2.dlpxdc.co	mawk	560	2 (https)	all 10 ✓

All 10 fields present on both: connections, cwnd, inbytes, outbytes, retranssegs, rtt, rwnd, suna, swnd, unsent. Prior to the fix: 0 rows in InfluxDB on the Azure engine. Removing the fflush() calls did not break streaming — -W interactive handles output flushing on its own.

Portability validation — official Ubuntu 24.04 base image

docker run --rm ubuntu:24.04 bash -c '
  dpkg -s mawk | grep -E "^(Status|Priority|Version)"
  echo "hello,world" | mawk -W interactive -F, "{print \$1, \$2}"
'

Output:

Status: install ok installed
Priority: required
Version: 1.3.4.20240123-1build1
hello world

On a completely unmodified ubuntu:24.04 image with no extras installed:

• mawk is Priority: required — guaranteed present on every Ubuntu system; cannot be removed without breaking apt
• /usr/bin/awk → mawk is the default alternative (gawk is not installed in the base image)
• mawk -W interactive works correctly

This confirms the fix is correct and portable across all current Delphix engine variants (AWS, Azure, GCP) running Ubuntu 24.04.

Related

• Fixes regression introduced by PR DLPX-98742 connstat-stats.sh: preserve connstat timestamps in InfluxDB #128 (DLPX-98742)
• Jira: DLPX-99899
• QA test: dlpx-qa-gate PR #9177 (engine_flux_templates_positive.py::test_tcp_stats_written_for_live_connections) is known_failure with DLPX-99899 and should be unblocked by this fix

Generated with Claude Code

[Copilot]

🟢 Approval recommended

The narrow buffering fix preserves the existing output contract, with reported live validation and no blocking findings.

No open issues 🎉

What changed in this PR

Restores streaming TCP metrics to Telegraf on engines using mawk, addressing the buffering regression introduced in PR #128.

Changes:

• Replaces stdbuf -oL awk with mawk -W interactive.
• Updates buffering comments without changing aggregation, timestamps, or CSV formatting.

File	Description
telegraf/​connstat-stats.sh	Enables mawk-native output flushing and updates explanatory comments.

_{Review effort: Balanced}

[prakashsurya]

Approving. The change is right and I reproduced it independently: under mawk 1.3.4 (the engine's version) the develop script prints nothing, and this PR's script streams correctly aggregated lines every interval. One thing to correct before merge: the description, the commit message and the new script comment all give the wrong cause, and the repo will keep that explanation.

The cause is mawk's input buffering, not its output. When mawk reads a pipe, it keeps calling read() until it has filled a 256 KB buffer (or hits EOF), and runs no print or fflush() before then. connstat writes only a few KB per interval and never exits, so nothing ever comes out. stdbuf -oL does take effect on mawk's stdout; it just can't help, because mawk never gets as far as writing. -W interactive fixes it because it switches stdin to line-by-line reads (it also unbuffers stdout). See man mawk: "sets unbuffered writes to stdout and line buffered reads from stdin".

Each of these takes a few seconds to check with strace:

# 1. Default mawk, fflush() after every print: reads keep filling the 256 KB buffer, and nothing is written
$ (while :; do echo a,b; sleep 1; done) | strace -e trace=read,write mawk -F, '{print; fflush()}' >/dev/null
read(0, "a,b\n", 262144)                = 4
read(0, "a,b\n", 262140)                = 4
read(0, "a,b\n", 262136)                = 4

# 2. -W interactive: each line is read, then written straight away
$ (while :; do echo a,b; sleep 1; done) | strace -e trace=read,write mawk -W interactive -F, '{print}' >/dev/null
read(0, "a,b\n", 4096)                  = 4
write(1, "a,b", 3)                      = 3
write(1, "\n", 1)                       = 1

# 3. stdbuf -oL does change mawk's stdout: one write per line, versus a single write without it
$ strace -f -e trace=write stdbuf -oL mawk 'BEGIN{print "a"; print "b"}' >/dev/null
write(1, "a\n", 2)                      = 2
write(1, "b\n", 2)                      = 2
$ strace -e trace=write mawk 'BEGIN{print "a"; print "b"}' >/dev/null
write(1, "a\nb\n", 4)                   = 4

Number 3 is the one that rules out "mawk bypasses libc stdio, so LD_PRELOAD can't reach it". It also explains the engine result in DLPX-99899, where {print; fflush()} gave 0 lines.

Requested before merge:

• The script comment: the suggestion inline below.
• The commit message and PR description: say mawk block-fills its stdin buffer, rather than "bypasses libc stdio" or "forces a flush after every print".

Optional:

• debian/control doesn't list mawk. It is Priority: required but not Essential, so adding it to Depends: would document the hard dependency.
• The two fflush() calls (L57, L85) now do nothing, since -W interactive already unbuffers stdout. They're harmless, so keep or drop them.

Also checked, no issues:

• Real Telegraf (1.40.2, the [[inputs.execd]] block from telegraf.base, with a stub connstat): this PR wrote a point every interval with the right tags, fields and timestamps. The old script under mawk wrote 0.
• Output unchanged: on the same input, the PR's script prints exactly what the old one does under gawk, /etc/services lookups included.
• Clean shutdown: stopping Telegraf leaves no connstat or mawk behind.
• Other collectors: none has the same bug. The rest use egrep --line-buffered or call commands directly.

[dbshah12]

Thanks for the thorough review and the independent strace repro — much appreciated.

Addressed in commit 4b14f5a (second commit on this branch):

• Script comment: applied the suggested wording verbatim; also switched to full /usr/bin/mawk path to match /usr/bin/connstat.
• Dropped both fflush() calls (formerly L57 and L85) — -W interactive already unbuffers stdout, so they were no-ops under the chosen fix.
• Added mawk to debian/control Depends: — documents the hard runtime dependency even though mawk is Priority: required on Ubuntu 24.04.

End-to-end verified on both engines before pushing (telegraf restarted, waited 5 min, queried InfluxDB):

Engine	awk resolves to	tcp_stats points in last 5 min	Series	Fields
dlpx-dev.dlpxdc.co	gawk	660	3 (https, ssh, ldap)	all 10 ✓
dlpx-qa-develop-qar-233997-766f97e2.dlpxdc.co	mawk	560	2 (https)	all 10 ✓

Note: the correction is captured in 4b14f5a's commit message, but the original 47b3907 commit message and the PR description above still carry the earlier (output-side) explanation. Happy to update the PR description now — let me know if you'd prefer that, or if the second-commit record is sufficient for the squash-merge.