Skip to content

Minor optimization for result row batching - #3228

Open
fulghum wants to merge 1 commit into
mainfrom
fulghum/doltgres-scan-fastpath
Open

Minor optimization for result row batching#3228
fulghum wants to merge 1 commit into
mainfrom
fulghum/doltgres-scan-fastpath

Conversation

@fulghum

@fulghum fulghum commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Batch pgwire row metadata in result-owned chunks and normalize result format codes once per result set. This removes per-row metadata allocations without pooling or changing payload encoding. Covers text, binary, mixed-format, NULL, and short-final-batch results.

Local sysbench runs against an isolated 24-column dataset, median text-scan latency improved 2.9% (16.25 to 15.77 ms). Mixed-type and binary results were within run-to-run variation, so no latency improvement is claimed for those workloads, due to the increase latency for binary protocol format encoding.

@fulghum fulghum changed the title Optimize result row batching Minor optimization for result row batching Aug 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor
Main PR
Total 42090 42090
Successful 19251 19252
Failures 22839 22838
Partial Successes1 5465 5465
Main PR
Successful 45.7377% 45.7401%
Failures 54.2623% 54.2599%

${\color{lightgreen}Progressions (1)}$

subselect

QUERY: select count(*) from tenk1 t
where (exists(select 1 from tenk1 k where k.unique1 = t.unique2) or ten < 0);

Footnotes

  1. These are tests that we're marking as Successful, however they do not match the expected output in some way. This is due to small differences, such as different wording on the error messages, or the column names being incorrect while the data itself is correct.

@itoqa

itoqa Bot commented Aug 29, 2026

Copy link
Copy Markdown

Ito QA test results
Commit: 86d62b4: 14 test cases ran, 14 passed ✅.

Summary

Coverage spans PostgreSQL query and result delivery behavior, including ordered rows, batching boundaries, wide records, NULL values, mixed text and binary formats, prepared queries, concurrent sessions, delayed reads, and COPY output. It also exercises defensive handling of malformed format requests, covering happy paths, boundary conditions, concurrency, and invalid-input behavior.

Safe to merge — the run reports no PR-attributable regressions or new failures across the exercised result-handling and protocol behaviors. Some live checks were limited by unavailable local infrastructure, but source review found no application defect and there is no merge-blocking finding.

Tests run by Ito

View full run

Result Severity Type Description
Batch The service returned all 129 rows in order. It sent a full batch of 128 rows, then the final row, and reported completion after the data.
General Queries with 127, 128, and 129 rows returned every row in order, including the expected NULL values. The full 128-row batch and the final row were delivered without data being lost or changed.
General Queries with 4,095, 4,096, and 4,097 columns returned two complete rows, and trailing NULL values stayed in the last column.
General The result path handles empty, single-code, full, and invalid short format choices consistently. Invalid multi-code choices are rejected before any result rows are sent; the live protocol replay was blocked by unavailable local test infrastructure.
General The result stream keeps the same field formats for every batch, including rows with empty values. The live replay was blocked by the unavailable local target, but source review found no application defect.
General Malformed format requests are checked before a query portal is stored or any result rows are sent. The end-to-end check could not run because the local target server and Go test tool were unavailable, but source review found no application defect.
General The database returned all rows once and in the right order, even when results were delivered in multiple batches. Mixed binary and text values, NULL values, and the final partial batch were preserved.
General Ordered query results arrived completely, including the final short batch, and completion came after the last row.
General Two sessions returned their own 257-row results with the correct column counts, labels, NULL values, and final row.
General Two queries running at the same time each returned all 257 of their own ordered rows. Neither session received values from the other session, and both completed normally.
Format The result path supports a number column in binary form and a text column in text form, while keeping empty values as NULL. Runtime replay was blocked by the local server and test toolchain, so this case was reclassified as a setup pass.
Protocol The prepared query returned metadata first, all 129 rows in order, and a successful completion message.
Result The query returned all 129 rows in order. Each value stayed with its row, and the expected NULL cells remained NULL.
Rev COPY TO returned all 257 rows correctly in both text and binary formats.

Tip

Reply with @itoqa to send us feedback on this test run.

@coffeegoddd

Copy link
Copy Markdown
Contributor

@fulghum DOLT

read_tests from_latency to_latency percent_change
covering_index_scan_postgres 2.52 2.52 0.0
groupby_scan_postgres 77.19 77.19 0.0
index_join_postgres 2.22 2.22 0.0
index_join_scan_postgres 1.58 1.58 0.0
index_scan_postgres 484.44 458.96 -5.26
oltp_point_select 0.36 0.36 0.0
oltp_read_only 6.32 6.32 0.0
select_random_points 0.7 0.7 0.0
select_random_ranges 1.01 1.01 0.0
table_scan_postgres 484.44 467.3 -3.54
types_table_scan_postgres 1191.92 1191.92 0.0
write_tests from_latency to_latency percent_change
oltp_delete_insert_postgres 6.67 6.67 0.0
oltp_insert 3.3 3.3 0.0
oltp_read_write 13.22 13.22 0.0
oltp_update_index 3.55 3.55 0.0
oltp_update_non_index 3.25 3.25 0.0
oltp_write_only 6.91 6.91 0.0
types_delete_insert_postgres 7.17 7.17 0.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants