Skip to content

[Bug]: CREATE INDEX ... USING diskann overwrites the heap's pg_class.reltuples with the indexed row count #283

Description

@zefir01

What happened?

ambuild() reports the number of tuples that went into the index as
IndexBuildResult.heap_tuples. That field is the table's live row count, not the
index's, and index_build() hands it straight to index_update_stats() for the
heap's pg_class row. So every CREATE INDEX ... USING diskann overwrites the
table's reltuples with the number of rows that index happens to contain.

pgvectorscale/src/access_method/build.rs:449-450 on main
(9d9851ce9966f01cdbbdc80e510d4938e3cd26bf), and 456-457 at tag 0.9.0:

    let mut result = unsafe { PgBox::<pg_sys::IndexBuildResult>::alloc0() };
    result.heap_tuples = ntuples as f64;
    result.index_tuples = ntuples as f64;

ntuples is what the build callbacks counted, i.e. the rows that reached the index.
index_tuples is right; heap_tuples is a different quantity and gets the same value.

PostgreSQL's two contracts, quoted from REL_17_10.
src/include/access/genam.h:30-34 says what the field means:

typedef struct IndexBuildResult
{
	double		heap_tuples;	/* # of tuples seen in parent table */
	double		index_tuples;	/* # of tuples inserted into index */
} IndexBuildResult;

src/backend/catalog/index.c:2766-2770, the index_update_stats() header comment,
says what is done with it:

 * hasindex: set relhasindex to this value
 * reltuples: if >= 0, set reltuples to this value; else no change
 *
 * If reltuples >= 0, relpages and relallvisible are also updated (using
 * RelationGetNumberOfBlocks() and visibilitymap_count()).

and src/backend/catalog/index.c:3100-3106, inside index_build(), is where the two
meet — note which relation each call names:

	index_update_stats(heapRelation,
					   true,
					   stats->heap_tuples);

	index_update_stats(indexRelation,
					   false,
					   stats->index_tuples);

For a full index over a live table the two counts coincide, which is why this is easy
to miss. For an index whose predicate skips rows they do not, and the table loses a
statistic that was never the index's to write. Nothing warns, and the value stays wrong
until the next ANALYZE or autovacuum.

The count heap_tuples wants already exists: it is the double that the table AM's
index_build_range_scan() returns, which both of pgvectorscale's scan wrappers discard
(pgrx's pg_sys::IndexBuildHeapScan drops it, and IndexBuildHeapScanParallel in
src/util/ports.rs returns ()).

Why this is about this access method and not about partial indexes

A partial btree and a partial hnsw over the same zero-row predicate leave reltuples
untouched: table_index_build_scan() counts a live tuple before it evaluates the
index predicate, and both of those AMs report the value it returned. Both controls are
in the script below and both hold at 2000.

Three consequences

1. The planner plans against a table it believes is empty. With reltuples = 0 and
relpages = 143 the density estimate is zero, so every row estimate over the table
collapses to the 1-row clamp. Measured below: rows=1 on a scan that returns 2000.

2. Autovacuum's analyze threshold collapses with it. relation_needs_vacanalyze()
reads this exact field — src/backend/postmaster/autovacuum.c:3065 is
reltuples = classForm->reltuples; and :3076 is
anlthresh = (float4) anl_base_thresh + anl_scale_factor * reltuples;. With the
documented defaults (autovacuum_analyze_threshold = 50,
autovacuum_analyze_scale_factor = 0.1) the threshold for this 2000-row table drops
from 50 + 0.1 * 2000 = 250 modifications to 50 + 0.1 * 0 = 50.

3. diskann disables its own parallel builds, which is what makes the defect
self-perpetuating.
This one is checkable inside your own code without a database:
ambuild() reads back the very statistic it overwrites. build.rs:315 on main:

    let heap_tuples = unsafe { heap_relation.rd_rel.as_ref().unwrap().reltuples as usize };

and build.rs:326:

            if heap_tuples >= min_vectors_for_parallel_build() {
                unsafe { (*index_info).ii_ParallelWorkers as usize }
            } else {
                0
            }

So one partial build that writes a value below diskann.min_vectors_for_parallel_build
(default 65536) makes every later diskann build on that table serial — silently, until
something re-ANALYZEs. The script below observes this in both directions through the
extension's own Parallel build with N workers notice: absent at reltuples = 0,
present at reltuples = 2000, same table, same settings, same statement.

pgvectorscale extension affected

0.9.0. main at 9d9851ce9966f01cdbbdc80e510d4938e3cd26bf still carries it — the line
numbers quoted above are main's.

PostgreSQL version used

17.10

What operating system did you use?

Ubuntu 22.04 x64, as shipped inside the timescale/timescaledb-ha:pg17 image.

What installation method did you use?

Docker

What platform did you run on?

On prem/Self-hosted

Relevant log output and stack trace

No error and no stack trace — that is the difficulty with this one. The full transcript
of the run described below, verbatim:

image:     timescale/timescaledb-ha:pg17
digest:    timescale/timescaledb-ha@sha256:eaaab826932c579d5d4e3a3479446da931cabb8703cf7caf1038eb14d40754fa
Pager usage is off.
CREATE EXTENSION
CREATE EXTENSION
   extname   | extversion 
-------------+------------
 vector      | 0.8.5
 vectorscale | 0.9.0
(2 rows)

                                                                version                                                                
---------------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.10 (Ubuntu 17.10-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0, 64-bit
(1 row)

psql:/tmp/heap-reltuples.sql:37: NOTICE:  table "probe" does not exist, skipping
DROP TABLE
CREATE TABLE
INSERT 0 2000
ANALYZE
### 1. baseline after ANALYZE -- expect reltuples = 2000
 relname | reltuples | relpages 
---------+-----------+----------
 probe   |      2000 |      143
(1 row)

### 2. partial btree over a predicate matching NO rows -- expect reltuples = 2000
CREATE INDEX
 relname | reltuples | relpages 
---------+-----------+----------
 probe   |      2000 |      143
(1 row)

### 3. partial hnsw over the same NO-row predicate -- expect reltuples = 2000
CREATE INDEX
 relname | reltuples | relpages 
---------+-----------+----------
 probe   |      2000 |      143
(1 row)

### 4. partial diskann over 1000 of 2000 rows -- OBSERVED reltuples = 1000
psql:/tmp/heap-reltuples.sql:65: NOTICE:  Starting index build with num_neighbors=-1, search_list_size=100, max_alpha=1.2, storage_layout=SbqCompression.
psql:/tmp/heap-reltuples.sql:65: NOTICE:  Indexed 1000 tuples
CREATE INDEX
 relname | reltuples | relpages 
---------+-----------+----------
 probe   |      1000 |      143
(1 row)

ANALYZE
### 5. re-ANALYZEd back to the truth -- reltuples = 2000 again
 relname | reltuples | relpages 
---------+-----------+----------
 probe   |      2000 |      143
(1 row)

### 6. partial diskann over a predicate matching NO rows -- OBSERVED reltuples = 0
psql:/tmp/heap-reltuples.sql:73: NOTICE:  Starting index build with num_neighbors=-1, search_list_size=100, max_alpha=1.2, storage_layout=SbqCompression.
psql:/tmp/heap-reltuples.sql:73: NOTICE:  Indexed 0 tuples
CREATE INDEX
 relname | reltuples | relpages 
---------+-----------+----------
 probe   |         0 |      143
(1 row)

### 7. the planner now estimates 1 row for a scan of a 2000-row table
                       QUERY PLAN                        
---------------------------------------------------------
 Seq Scan on probe  (cost=0.00..143.00 rows=1 width=524)
   Filter: (id > 0)
(2 rows)

 actual_live_rows 
------------------
             2000
(1 row)

### 8. reltuples is still 0 here; expect NO "Parallel build" notice
SET
SET
SET
psql:/tmp/heap-reltuples.sql:96: NOTICE:  Starting index build with num_neighbors=-1, search_list_size=100, max_alpha=1.2, storage_layout=SbqCompression.
psql:/tmp/heap-reltuples.sql:96: NOTICE:  Indexed 2000 tuples
CREATE INDEX
 relname | reltuples 
---------+-----------
 probe   |      2000
(1 row)

ANALYZE
### 9. same table, same settings, reltuples = 2000 -- expect the notice
psql:/tmp/heap-reltuples.sql:101: NOTICE:  Starting index build with num_neighbors=-1, search_list_size=100, max_alpha=1.2, storage_layout=SbqCompression.
psql:/tmp/heap-reltuples.sql:101: NOTICE:  Parallel build with 2 workers
psql:/tmp/heap-reltuples.sql:101: NOTICE:  Indexed 474 tuples
psql:/tmp/heap-reltuples.sql:101: NOTICE:  Indexed 1526 tuples
CREATE INDEX
 relname | reltuples 
---------+-----------
 probe   |      2000
(1 row)

### 10. per-relation reltuples: the indexes are right, the heap was not
        relname         | kind  | reltuples 
------------------------+-------+-----------
 probe                  | table |      2000
 probe_btree_none       | index |         0
 probe_diskann_half     | index |      1000
 probe_diskann_none     | index |         0
 probe_diskann_parallel | index |      2000
 probe_diskann_serial   | index |      2000
 probe_hnsw_none        | index |         0
(7 rows)

DROP TABLE

How can we reproduce the bug?

The public timescale/timescaledb-ha:pg17 image already carries everything needed —
pgvector 0.8.5 and pgvectorscale 0.9.0 on PostgreSQL 17.10 — so the reproduction needs
no build and nothing else installed. It creates and drops a table called probe, so do
not point it at a server that matters:

docker run -d --name vs-repro --network none -e POSTGRES_PASSWORD=repro \
    timescale/timescaledb-ha:pg17
until docker exec vs-repro pg_isready -q; do sleep 1; done

docker cp heap-reltuples.sql vs-repro:/tmp/
docker exec -u postgres vs-repro psql -X -f /tmp/heap-reltuples.sql

docker rm -f vs-repro

heap-reltuples.sql:

\set ON_ERROR_STOP on
\pset pager off

CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS vectorscale;

SELECT extname, extversion FROM pg_extension
 WHERE extname IN ('vector', 'vectorscale') ORDER BY extname;
SELECT version();

-- ---------------------------------------------------------------- the fixture
DROP TABLE IF EXISTS probe CASCADE;
CREATE TABLE probe (id int, v vector(128));

INSERT INTO probe (id, v)
SELECT g, (SELECT array_agg(sin(g * d)::real) FROM generate_series(1, 128) d)::vector
  FROM generate_series(1, 2000) g;

ANALYZE probe;

\echo '### 1. baseline after ANALYZE -- expect reltuples = 2000'
SELECT relname, reltuples, relpages FROM pg_class WHERE relname = 'probe';

-- ------------------------------------------------------- control: btree
-- A partial btree over a predicate matching no rows.  table_index_build_scan()
-- counts a live tuple BEFORE it evaluates the index predicate, and btree
-- reports the value it returned, so the heap's statistic is untouched.
\echo '### 2. partial btree over a predicate matching NO rows -- expect reltuples = 2000'
CREATE INDEX probe_btree_none ON probe (id) WHERE id > 100000;
SELECT relname, reltuples, relpages FROM pg_class WHERE relname = 'probe';

-- -------------------------------------------------------- control: hnsw
-- Same predicate, same column, pgvector's own vector access method.
\echo '### 3. partial hnsw over the same NO-row predicate -- expect reltuples = 2000'
CREATE INDEX probe_hnsw_none ON probe USING hnsw (v vector_l2_ops) WHERE id > 100000;
SELECT relname, reltuples, relpages FROM pg_class WHERE relname = 'probe';

-- ------------------------------------------------------------- the defect
\echo '### 4. partial diskann over 1000 of 2000 rows -- OBSERVED reltuples = 1000'
CREATE INDEX probe_diskann_half ON probe USING diskann (v) WHERE id > 1000;
SELECT relname, reltuples, relpages FROM pg_class WHERE relname = 'probe';

ANALYZE probe;
\echo '### 5. re-ANALYZEd back to the truth -- reltuples = 2000 again'
SELECT relname, reltuples, relpages FROM pg_class WHERE relname = 'probe';

\echo '### 6. partial diskann over a predicate matching NO rows -- OBSERVED reltuples = 0'
CREATE INDEX probe_diskann_none ON probe USING diskann (v) WHERE id > 100000;
SELECT relname, reltuples, relpages FROM pg_class WHERE relname = 'probe';

-- ------------------------------------------------------- consequence: planner
-- reltuples = 0 with relpages > 0 makes the density estimate zero, so every
-- row estimate over this table collapses to the 1-row clamp.
\echo '### 7. the planner now estimates 1 row for a scan of a 2000-row table'
EXPLAIN (COSTS ON, TIMING OFF, SUMMARY OFF) SELECT * FROM probe WHERE id > 0;
SELECT count(*) AS actual_live_rows FROM probe;

-- ------------------------- consequence: diskann disables its own parallel build
-- ambuild() reads heap_relation.rd_rel.reltuples and compares it against
-- diskann.min_vectors_for_parallel_build to decide whether to build in
-- parallel, so a build that clobbered the statistic makes every later build on
-- that table serial until something re-ANALYZEs it.  The threshold is lowered
-- here only so that a 2000-row fixture can cross it; the default is 65536.
--
-- Both builds below are FULL indexes, so neither leaves a wrong reltuples --
-- which is the other half of why the defect is easy to miss.
\echo '### 8. reltuples is still 0 here; expect NO "Parallel build" notice'
SET min_parallel_table_scan_size = 0;
SET max_parallel_maintenance_workers = 2;
SET diskann.min_vectors_for_parallel_build = 1000;
CREATE INDEX probe_diskann_serial ON probe USING diskann (v);
SELECT relname, reltuples FROM pg_class WHERE relname = 'probe';

ANALYZE probe;
\echo '### 9. same table, same settings, reltuples = 2000 -- expect the notice'
CREATE INDEX probe_diskann_parallel ON probe USING diskann (v);
SELECT relname, reltuples FROM pg_class WHERE relname = 'probe';

-- ------------------------------------ what the index's own statistic should be
-- index_tuples is the field for the indexed count, and it is correct: each
-- index's own reltuples matches what it holds.  Only the heap's row is wrong.
\echo '### 10. per-relation reltuples: the indexes are right, the heap was not'
SELECT c.relname,
       CASE c.relkind WHEN 'r' THEN 'table' WHEN 'i' THEN 'index' END AS kind,
       c.reltuples
  FROM pg_class c
 WHERE c.relname LIKE 'probe%'
 ORDER BY c.relkind DESC, c.relname;

DROP TABLE probe CASCADE;

Proposed fix

Report the live heap tuples in heap_tuples and keep the indexed count in
index_tuples, which is the field for it. The value is the one the heap scan already
computes, so nothing extra is scanned or counted:

  • src/util/ports.rs gains a serial IndexBuildHeapScan that returns the f64 the
    pgrx wrapper throws away (pgrx's own wrapper cannot be fixed from here), and
    IndexBuildHeapScanParallel stops returning ();
  • each parallel worker adds its share into a new ParallelBuildState field — the same
    accumulation nbtree performs on btshared->reltuples under its spinlock, done here
    with a compare-exchange on the f64's bits so no lock is added to the struct;
  • ambuild() assigns the two counts separately.

The Indexed N tuples notices and every index's own pg_class.reltuples are unchanged
by this; only the heap's row moves. Re-running the script above against a build carrying
the patch gives reltuples = 2000 at every step — both partial diskann builds, serial
and parallel — rows=2000 from the same EXPLAIN, and the same per-index values
0 / 1000 / 0 / 2000 / 2000 / 0.

Applies cleanly to 0.9.0 and to main at 9d9851ce (offset only, no conflicts).
Built and exercised on 0.9.0 with pgrx 0.16.1 — the version Cargo.toml pins — under
PostgreSQL 17.10 and pgvector 0.8.5.

The patch — git format-patch output, so git am takes it directly
From 827294b34e95864edb154dc01c0ec13ebfcda9b0 Mon Sep 17 00:00:00 2001
From: Peter Stukalov <peter.stukalov01@gmail.com>
Date: Mon, 7 Sep 2026 06:32:12 +0000
Subject: [PATCH] Fix ambuild overwriting the heap's reltuples with the indexed
 tuple count

`IndexBuildResult.heap_tuples` is "# of tuples seen in parent table"
(`access/genam.h`), and `index_build()` hands it straight to
`index_update_stats()` for the *heap's* pg_class row:

	index_update_stats(heapRelation,
					   true,
					   stats->heap_tuples);

whose contract is, in `src/backend/catalog/index.c`:

	 * hasindex: set relhasindex to this value
	 * reltuples: if >= 0, set reltuples to this value; else no change
	 *
	 * If reltuples >= 0, relpages and relallvisible are also updated

`ambuild()` puts the number of tuples that went *into the index* there, so
every diskann build overwrites the table's reltuples with its own row count.
For a full index over a live table the two numbers coincide, which is why
this is easy to miss. For an index that covers a subset of the table they do
not, and the heap loses a statistic that is not the index's to write:

  CREATE TABLE probe (id int, v vector(128));
  INSERT INTO probe SELECT g, ... FROM generate_series(1, 2000) g;
  ANALYZE probe;                                    -- reltuples = 2000

  CREATE INDEX probe_half ON probe USING diskann (v) WHERE id > 1000;
                                                    -- reltuples = 1000
  CREATE INDEX probe_none ON probe USING diskann (v) WHERE id > 100000;
                                                    -- reltuples = 0

A partial btree or hnsw index over the same table leaves reltuples alone:
`table_index_build_scan()` counts a live tuple before it evaluates the index
predicate, and both of those AMs report the value it returned.

reltuples = 0 is worse than a stale estimate. The planner then plans against
a table it believes is empty, and diskann disables its own parallel builds
for every index created afterwards, because `ambuild()` reads
`heap_relation.rd_rel.reltuples` to compare against
`diskann.min_vectors_for_parallel_build`. Nothing warns, and the value stays
wrong until the next ANALYZE or autovacuum.

The count the field wants is the one the heap scan already computes and both
of pgvectorscale's scan wrappers throw away:

  * `util::ports::IndexBuildHeapScan` returns the double that the table AM's
    `index_build_range_scan()` produced. pgrx's `pg_sys::IndexBuildHeapScan`
    discards it and cannot be fixed from here, so this is a copy of that
    wrapper which does not discard it;
  * `IndexBuildHeapScanParallel` returns each worker's share, and the workers
    accumulate it into a new field of the shared `ParallelBuildState` -- the
    same accumulation nbtree performs on `btshared->reltuples` under its
    spinlock;
  * `ambuild()` reports the live heap tuples in `heap_tuples` and keeps the
    indexed count in `index_tuples`, which is the field for it.

With the patch the reproduction above leaves reltuples at 2000 for both
partial builds, serial and parallel (`diskann.force_parallel_workers = 2`),
while the "Indexed 1000 tuples" / "Indexed 0 tuples" notices and the
indexes' own pg_class.reltuples are unchanged.

Tested on PostgreSQL 17.10 with pgvector 0.8.5.
---
 pgvectorscale/src/access_method/build.rs | 134 ++++++++++++++++++-----
 pgvectorscale/src/util/ports.rs          |  41 ++++++-
 2 files changed, 146 insertions(+), 29 deletions(-)

diff --git a/pgvectorscale/src/access_method/build.rs b/pgvectorscale/src/access_method/build.rs
index 0aea049..af712a7 100644
--- a/pgvectorscale/src/access_method/build.rs
+++ b/pgvectorscale/src/access_method/build.rs
@@ -1,4 +1,4 @@
-use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
+use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
 use std::time::Instant;
 
 use pg_sys::{FunctionCall0Coll, InvalidOid};
@@ -18,7 +18,7 @@ use crate::util::ports::acquire_index_lock;
 
 use crate::access_method::DISKANN_DISTANCE_TYPE_PROC;
 use crate::util::page::PageType;
-use crate::util::ports::IndexBuildHeapScanParallel;
+use crate::util::ports::{IndexBuildHeapScan, IndexBuildHeapScanParallel};
 use crate::util::tape::Tape;
 use crate::util::*;
 
@@ -223,11 +223,54 @@ struct ParallelSharedParams {
 #[cfg_attr(not(feature = "build_parallel"), allow(dead_code))]
 struct ParallelBuildState {
     ntuples: AtomicUsize,
+    /// Bits of an `f64`: the live heap tuples seen by all workers' scans, summed.
+    /// This is a different quantity from `ntuples`, which counts only the tuples
+    /// that went into the index.
+    live_heap_tuples: AtomicU64,
     start_nodes_initialized: AtomicBool,
     initializing_worker_done: AtomicBool,
     initialization_cv: ConditionVariable,
 }
 
+#[cfg_attr(not(feature = "build_parallel"), allow(dead_code))]
+impl ParallelBuildState {
+    /// Add one worker's live heap tuple count to the shared total.  nbtree does
+    /// the same accumulation under a spinlock (`btshared->reltuples +=
+    /// reltuples`); an atomic compare-exchange on the `f64`'s bits keeps the
+    /// value exact without adding a lock to this struct.
+    fn add_live_heap_tuples(&self, tuples: f64) {
+        let mut current = self.live_heap_tuples.load(Ordering::Relaxed);
+        loop {
+            let total = (f64::from_bits(current) + tuples).to_bits();
+            match self.live_heap_tuples.compare_exchange_weak(
+                current,
+                total,
+                Ordering::Relaxed,
+                Ordering::Relaxed,
+            ) {
+                Ok(_) => return,
+                Err(observed) => current = observed,
+            }
+        }
+    }
+
+    fn live_heap_tuples(&self) -> f64 {
+        f64::from_bits(self.live_heap_tuples.load(Ordering::Relaxed))
+    }
+}
+
+/// What a heap scan for an index build produced.
+///
+/// The two counts differ for a partial index, and for any index whose predicate
+/// or expression skips rows: `index_tuples` counts what was indexed, while
+/// `live_heap_tuples` counts every live row the scan walked past.  Postgres
+/// writes the latter into the *heap's* `pg_class.reltuples`, so conflating them
+/// corrupts the table's statistics.
+struct HeapScanResult {
+    index_tuples: usize,
+    live_heap_tuples: f64,
+}
+
 /// Status data for parallel index builds, shared among all parallel workers.
 #[derive(Debug)]
 #[cfg_attr(not(feature = "build_parallel"), allow(dead_code))]
@@ -385,6 +428,7 @@ pub extern "C-unwind" fn ambuild(
                     },
                     build_state: ParallelBuildState {
                         ntuples: AtomicUsize::new(0),
+                        live_heap_tuples: AtomicU64::new(0.0f64.to_bits()),
                         start_nodes_initialized: AtomicBool::new(false),
                         initializing_worker_done: AtomicBool::new(false),
                         initialization_cv: std::mem::zeroed(), // Will be initialized below
@@ -427,7 +471,7 @@ pub extern "C-unwind" fn ambuild(
         None
     };
 
-    let ntuples = if let Some(ParallelData { pcxt, snapshot }) = parallel_data {
+    let scan_result = if let Some(ParallelData { pcxt, snapshot }) = parallel_data {
         unsafe {
             pg_sys::WaitForParallelWorkersToFinish(pcxt);
             let parallel_shared: *mut ParallelShared =
@@ -437,8 +481,12 @@ pub extern "C-unwind" fn ambuild(
                 .build_state
                 .ntuples
                 .load(Ordering::Relaxed);
+            let live_heap_tuples = (*parallel_shared).build_state.live_heap_tuples();
             parallel::cleanup_parallel_context(pcxt, snapshot);
-            ntuples
+            HeapScanResult {
+                index_tuples: ntuples,
+                live_heap_tuples,
+            }
         }
     } else {
         do_heap_scan(
@@ -453,8 +501,12 @@ pub extern "C-unwind" fn ambuild(
     };
 
     let mut result = unsafe { PgBox::<pg_sys::IndexBuildResult>::alloc0() };
-    result.heap_tuples = ntuples as f64;
-    result.index_tuples = ntuples as f64;
+    // `heap_tuples` is the number of tuples seen in the parent table, and
+    // `index_build` hands it straight to `index_update_stats` for the *heap's*
+    // pg_class row.  It must be the scan's live tuple count, never the number of
+    // rows this index happens to contain.
+    result.heap_tuples = scan_result.live_heap_tuples;
+    result.index_tuples = scan_result.index_tuples as f64;
 
     result.into_pg()
 }
@@ -594,7 +646,8 @@ fn maybe_train_quantizer(
             };
 
             unsafe {
-                pg_sys::IndexBuildHeapScan(
+                // The quantizer only needs the sample, not the tuple count.
+                IndexBuildHeapScan(
                     heap_relation.as_ptr(),
                     index_relation.as_ptr(),
                     index_info,
@@ -723,7 +776,7 @@ fn do_heap_scan(
     mut write_stats: WriteStats,
     parallel_build_info: Option<ParallelBuildInfo>,
     worker_count: usize,
-) -> usize {
+) -> HeapScanResult {
     unsafe {
         pgstat_progress_update_param(PROGRESS_CREATE_IDX_SUBPHASE, BUILD_PHASE_BUILDING_GRAPH);
     }
@@ -762,7 +815,7 @@ fn do_heap_scan(
                 );
                 let mut state = StorageBuildStateParallel::Plain(&mut plain, &mut bs);
 
-                unsafe {
+                let live_heap_tuples = unsafe {
                     IndexBuildHeapScanParallel(
                         heap_relation.as_ptr(),
                         index_relation.as_ptr(),
@@ -770,12 +823,21 @@ fn do_heap_scan(
                         Some(build_callback_parallel),
                         &mut state,
                         parallel_info.tablescandesc,
-                    );
-                }
+                    )
+                };
+                shared_state.build_state.add_live_heap_tuples(live_heap_tuples);
 
                 // In parallel mode, nodes are finalized during insertion via streaming
                 // Just need to handle any remaining cached nodes and update meta page
-                finalize_remaining_parallel_nodes(&mut plain, bs, index_relation, write_stats)
+                HeapScanResult {
+                    index_tuples: finalize_remaining_parallel_nodes(
+                        &mut plain,
+                        bs,
+                        index_relation,
+                        write_stats,
+                    ),
+                    live_heap_tuples,
+                }
             }
             StorageType::SbqCompression => {
                 let mut bq = unsafe {
@@ -797,7 +859,7 @@ fn do_heap_scan(
                 );
                 let mut state = StorageBuildStateParallel::SbqSpeedup(&mut bq, &mut bs);
 
-                unsafe {
+                let live_heap_tuples = unsafe {
                     IndexBuildHeapScanParallel(
                         heap_relation.as_ptr(),
                         index_relation.as_ptr(),
@@ -805,8 +867,9 @@ fn do_heap_scan(
                         Some(build_callback_parallel),
                         &mut state,
                         parallel_info.tablescandesc,
-                    );
-                }
+                    )
+                };
+                shared_state.build_state.add_live_heap_tuples(live_heap_tuples);
 
                 unsafe {
                     pgstat_progress_update_param(
@@ -817,7 +880,15 @@ fn do_heap_scan(
 
                 // In parallel mode, nodes are finalized during insertion via streaming
                 // Just need to handle any remaining cached nodes and update meta page
-                finalize_remaining_parallel_nodes(&mut bq, bs, index_relation, write_stats)
+                HeapScanResult {
+                    index_tuples: finalize_remaining_parallel_nodes(
+                        &mut bq,
+                        bs,
+                        index_relation,
+                        write_stats,
+                    ),
+                    live_heap_tuples,
+                }
             }
         }
     } else {
@@ -842,17 +913,25 @@ fn do_heap_scan(
                 let mut bs = BuildState::new(index_relation, graph, page_type);
                 let mut state = StorageBuildState::Plain(&mut plain, &mut bs);
 
-                unsafe {
-                    pg_sys::IndexBuildHeapScan(
+                let live_heap_tuples = unsafe {
+                    IndexBuildHeapScan(
                         heap_relation.as_ptr(),
                         index_relation.as_ptr(),
                         index_info,
                         Some(build_callback),
                         &mut state,
-                    );
-                }
+                    )
+                };
 
-                finalize_index_build(&mut plain, bs, index_relation, write_stats)
+                HeapScanResult {
+                    index_tuples: finalize_index_build(
+                        &mut plain,
+                        bs,
+                        index_relation,
+                        write_stats,
+                    ),
+                    live_heap_tuples,
+                }
             }
             StorageType::SbqCompression => {
                 let mut bq = unsafe {
@@ -868,15 +947,15 @@ fn do_heap_scan(
                 let mut bs = BuildState::new(index_relation, graph, page_type);
                 let mut state = StorageBuildState::SbqSpeedup(&mut bq, &mut bs);
 
-                unsafe {
-                    pg_sys::IndexBuildHeapScan(
+                let live_heap_tuples = unsafe {
+                    IndexBuildHeapScan(
                         heap_relation.as_ptr(),
                         index_relation.as_ptr(),
                         index_info,
                         Some(build_callback),
                         &mut state,
-                    );
-                }
+                    )
+                };
 
                 unsafe {
                     pgstat_progress_update_param(
@@ -885,7 +964,10 @@ fn do_heap_scan(
                     );
                 }
 
-                finalize_index_build(&mut bq, bs, index_relation, write_stats)
+                HeapScanResult {
+                    index_tuples: finalize_index_build(&mut bq, bs, index_relation, write_stats),
+                    live_heap_tuples,
+                }
             }
         }
     }
diff --git a/pgvectorscale/src/util/ports.rs b/pgvectorscale/src/util/ports.rs
index 2531d47..1f1d956 100644
--- a/pgvectorscale/src/util/ports.rs
+++ b/pgvectorscale/src/util/ports.rs
@@ -178,7 +178,42 @@ pub fn buffer_align(len: usize) -> usize {
     }
 }
 
-/// Custom IndexBuildHeapScan that uses parallel table scan descriptor
+/// Scan the heap for an index build, returning the number of live heap tuples the
+/// scan saw -- the `double` that `IndexBuildResult.heap_tuples` is defined to
+/// carry ("# of tuples seen in parent table", `access/genam.h`).
+///
+/// pgrx's own `pg_sys::IndexBuildHeapScan` wrapper drops that return value, so it
+/// cannot be used where the count is needed.
+#[allow(non_snake_case)]
+pub unsafe fn IndexBuildHeapScan<T>(
+    heap_relation: pg_sys::Relation,
+    index_relation: pg_sys::Relation,
+    index_info: *mut pg_sys::IndexInfo,
+    build_callback: pg_sys::IndexBuildCallback,
+    build_callback_state: *mut T,
+) -> f64 {
+    let heap_relation_ref = heap_relation.as_ref().unwrap();
+    let table_am = heap_relation_ref.rd_tableam.as_ref().unwrap();
+
+    table_am.index_build_range_scan.unwrap()(
+        heap_relation,
+        index_relation,
+        index_info,
+        true,                       // allow_sync
+        false,                      // anyvisible
+        true,                       // progress
+        0,                          // start_blockno
+        pg_sys::InvalidBlockNumber, // end_blockno
+        build_callback,
+        build_callback_state as *mut std::os::raw::c_void,
+        std::ptr::null_mut(),
+    )
+}
+
+/// Custom IndexBuildHeapScan that uses parallel table scan descriptor.
+///
+/// Returns the number of live heap tuples this worker's share of the scan saw;
+/// the caller is responsible for summing the workers' counts.
 #[allow(non_snake_case)]
 pub unsafe fn IndexBuildHeapScanParallel<T>(
     heap_relation: pg_sys::Relation,
@@ -187,7 +222,7 @@ pub unsafe fn IndexBuildHeapScanParallel<T>(
     build_callback: pg_sys::IndexBuildCallback,
     build_callback_state: *mut T,
     tablescandesc: *mut pg_sys::ParallelTableScanDescData,
-) {
+) -> f64 {
     let scan = pg_sys::table_beginscan_parallel(heap_relation, tablescandesc);
 
     let heap_relation_ref = heap_relation.as_ref().unwrap();
@@ -205,7 +240,7 @@ pub unsafe fn IndexBuildHeapScanParallel<T>(
         build_callback,
         build_callback_state as *mut std::os::raw::c_void,
         scan,
-    );
+    )
 }
 
 /// Is a snapshot MVCC-safe? (This should really be a part of pgrx)
-- 
2.34.1

Are you going to work on the bugfix?

The fix is above and it is yours to take as-is. I am opening this as an issue rather than
a pull request deliberately, on three things CONTRIBUTING.md says: that the issues page
is "the best place to discuss your proposed improvement (and its implementation) with the
core development team"; that a code contribution needs a signed CLA; and that a
non-trivial change should come with a test-suite addition and a passing full suite, which
this patch does not yet have — I have exercised it against a real server rather than
through cargo pgrx test.

Happy to open the PR with a #[pg_test] covering the partial-index case if you would
like it in that form — just say so. Equally happy for you to write it differently; the
part I would ask you not to lose is that heap_tuples and index_tuples are two
different quantities.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions