[hdq66666]

Summary

Add a fallback for NVLink bandwidth reporting when DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL is unavailable.

I observed that field 449 reports N/A on Tesla V100 systems, while the NVLink TX/RX profiling fields still provide valid values. This change has been tested successfully on V100.

Changes

• Collect NVLink TX/RX profiling fields 1011/1012.
• Prefer field 449 when available.
• Fall back to TX + RX when field 449 is N/A.
• Add tests for parsing and fallback behavior.

[mmshad]

@hdq66666, Thanks for this, and for tracking down that field 449 reports N/A on V100. That reproduces on our side too: 449 comes back N/A on our V100 nodes while A100 and H100 both report a value.

I pushed a commit on top of yours. What needed fixing:

• Clamp the synthesized value. The mapped 449 path goes through convert(), which applies max(0.0, ...). The fallback summed raw values, so a negative reading raised TranslateError out of the live loop and took the dashboard down rather than blanking one field.
• Require both directions. Summing with "tx or 0.0" meant a one-sided reading was reported as the full aggregate, silently halving the number. It now reports nothing instead of something wrong.
• Route through convert() rather than float(), so the fallback follows the same unit discipline as every other field.

Also added tests for the idle-versus-missing case, one-sided readings and negative input, plus a changelog entry, and merged current main in since 0.5.0 landed after you opened this.

Validated on hardware. I generated NVLink traffic on an idle H100 and ran the same capture through both paths:

• field 449: 129.93 GB/s
• TX plus RX fallback: 136.23 GB/s

The two agree to about 5 percent, with the profiling counters reading slightly high because they include protocol headers. Close enough for a fallback, but worth knowing they are not interchangeable.

One thing I checked because it concerned me: going from 10 to 12 profiling fields does not starve the others. SM, tensor and DRAM fill rates were unchanged on both H100 and H200.

[mmshad]

@Naeemkh , this PR addresses a real issue in KempnerPulse. I fixed a few issues, and it's ready for your final review before merging.

[Naeemkh]

Thank you, This is a real gap and the fix is in the right layer. I confirmed fields 1011/1012 are in DCGM profiling group A.0, the same group as the 1001–1010 we already collect, so this adds no multiplexing pressure at the 100 ms floor. Tests pass locally (156 → 163). Three things before I approve:

1. Keep an "NVLink unavailable" row in the fixture. GPU 0 tick 2 was the only row where the canonical aggregate came out None; giving it valid TX/RX means no fixture row exercises that path any more. The comment at tests/integration/test_pipeline.py:72 ("gpu0 NVLink was N/A in tick 2") is now inaccurate, and its assertion is weak enough that nothing failed. Please leave TX/RX as N/A on one row, or add a row.
2. Fix the two doc lines this contradicts. docs/metrics.md:75-79 calls field 449 a "monotonic counter converted to rate" and docs/schema.md:100 labels the canonical field "bytes/s (differenced)". Neither matches the code, which treats 449 as an MB/s gauge and deliberately does not difference it. I checked our H200 captures: the values fluctuate and return to zero, non-monotonic on every GPU, so the code is right and the docs are wrong. Please correct both lines and add the new fields plus the fallback rule to docs/metrics.md.
3. Scope the CHANGELOG claim to the dcgmi backend. "Those two fields are now collected" holds for --backend dcgm; on the prometheus path, dcgm-exporter's shipped counters CSVs don't include 1011/1012, so operators have to add them. Worth a line in docs/guide/backends.md.

[hdq66666]

Addressed all three review points in 76c4a49: restored an end-to-end NVLink-unavailable fixture path with explicit canonical and CSV assertions; corrected field 449 to an MB/s gauge and documented fields 1011/1012 plus the fallback rule; and scoped collection to the direct dcgm backend while documenting the dcgm-exporter counters CSV requirement. Locally, all 163 tests pass and the strict Sphinx build succeeds.