Skip to content

perf: widen AVX2 percentile scan to 16 int64/iter (vector accumulator)#138

Open
fcostaoliveira wants to merge 1 commit into
HdrHistogram:mainfrom
fcostaoliveira:perf/avx2-percentile-scan-widen16
Open

perf: widen AVX2 percentile scan to 16 int64/iter (vector accumulator)#138
fcostaoliveira wants to merge 1 commit into
HdrHistogram:mainfrom
fcostaoliveira:perf/avx2-percentile-scan-widen16

Conversation

@fcostaoliveira

Copy link
Copy Markdown
Contributor

Summary

Widen the AVX2 percentile scan (get_value_from_idx_up_to_count_avx2, added in #134) from
4 → 16 int64 per iteration using a vector accumulator.

The current loop processes 4 counts per iteration and, every iteration, does a horizontal
reduction plus two _mm_extract_epi64 (vector→GPR moves) plus the running-total early-exit
branch. Those are the expensive parts. This PR accumulates 16 counts per iteration (4×256-bit
loads summed in a __m256i) and reduces to a scalar block sum once per 16 elements, so the
GPR extracts and the branch run 4× less often. The four loads and three vpaddq pipeline on
the load/ALU ports.

Only the block granularity changes:

  • the scalar fallback (#ifndef HDR_HAS_AVX2_DISPATCH) is untouched;
  • the uint64_t chunk-sum overflow hardening is preserved;
  • the per-element fine walk (taken only for the single block that crosses the target) and the
    scalar remainder tail are unchanged;
  • results are bit-identical to the previous scan.

Benchmark

test/hdr_percentile_benchhdr_value_at_percentile throughput, best of 20 runs after 3
warmups, pinned to one core, measured base vs patch back-to-back in the same session on an
Intel Xeon Gold 6248 (Cascade Lake):

Compiler Base (main) This PR Δ
gcc 11.4 0.16 M q/s 0.38 M q/s +137%
clang 14.0 0.18 M q/s 0.44 M q/s +144%

The benchmark's sink accumulator is byte-identical between base and patch on both compilers
(17401860284404480), i.e. every percentile query returns exactly the same value as before.

Steps to reproduce

# base (main) and patch, each:
cmake -S . -B build -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DHDR_HISTOGRAM_BUILD_PROGRAMS=ON -DHDR_HISTOGRAM_BUILD_BENCHMARK=ON
cmake --build build -j
taskset -c 8 ./build/test/hdr_percentile_bench     # compare M queries/sec + sink

Correctness

  • ctest green (gcc and clang).
  • ASan + UBSan clean; the 16-wide loop only runs for idx < (counts_len & ~15), so the widest
    load counts[idx..idx+15] stays in bounds; the remainder is handled by the existing scalar tail.
  • Identical hdr_value_at_percentile output verified via the byte-identical benchmark sink above.

Relationship to #137

This optimizes the existing (#134) AVX2 path and is independent of #137. If you prefer #137's
portable scalar block-sum (dropping the AVX2 dispatch), I'm happy to re-target — I can send the
same 16-wide vector-accumulator idea rebased on that, or the equivalent widening of the portable
block loop. Whichever direction you'd like, just say the word. Note #137 also restores the
normalizing_index_offset-aware fallback for decoded histograms; that behavior is orthogonal to
this change (this PR does not alter the direct-counts[] read introduced in #134).

…ator

get_value_from_idx_up_to_count_avx2 summed 4 int64/iter and did a horizontal
reduction + 2x _mm_extract_epi64 + target-cross branch every 4 elements. Accumulate
16 int64/iter (4x256) in a vector register and reduce to a scalar block sum once per
16, so the costly GPR extracts and the early-exit branch run 4x less often. Scalar
fallback and uint64 overflow hardening unchanged; percentile results bit-identical.

clx1 (Cascade Lake), core-pinned, same-session A/B: hdr_value_at_percentile
+137% (gcc 0.16->0.38 Mq/s) / +144% (clang 0.18->0.44 Mq/s). Read sink byte-identical.
fcostaoliveira pushed a commit to redis-performance/hdr-agent-workspace that referenced this pull request Jul 1, 2026
…hmark data

Opened HdrHistogram/HdrHistogram_c#138 (fcostaoliveira:
perf/avx2-percentile-scan-widen16 -> HdrHistogram:main, +15/-7, MERGEABLE). Body carries
the clx1 same-session A/B table (read +137% gcc / +144% clang), repro via hdr_percentile_bench,
correctness (bit-identical sink, ctest, ASan/UBSan), and the #137 relationship. Logs synced
(EXPERIMENTS/SUMMARY/README/memory).
@fcostaoliveira

Copy link
Copy Markdown
Contributor Author

⚠️ Conflict heads-up: this enhances the AVX2 percentile scan, whereas my open #137 proposes to remove the AVX2 dispatch entirely (portable scalar block-sum). #138/#139 and #137 are two directions for the same read path — they conflict. Flagging for coordination; happy to pick one.

fcostaoliveira pushed a commit to fcostaoliveira/HdrHistogram_c that referenced this pull request Jul 2, 2026
The widened AVX2 percentile scan is memory-load-latency bound over the ~10s-of-KB
counts[] array. Prefetch 4 iterations (512 B) ahead with _MM_HINT_T0 to hide L2/L3
latency. Read throughput (hdr_value_at_percentile), same-session core-pinned A/B:
  Cascade Lake (Xeon Gold 6248): gcc +8%, clang neutral
  Granite Rapids:                gcc +7.7%, clang +5.7%
Write path unaffected (control flat on both); percentile results bit-identical.

Stacked on perf/avx2-percentile-scan-widen16 (PR HdrHistogram#138).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant