Native support for stddev and variance related window functions

[fulghum]

Fixed a bug where STDDEV_POP, STDDEV_SAMP, VAR_POP, VAR_SAMP, and their variance and stddev aliases would crash the server when used as window functions over integer columns. They now return correct, Postgres-compatible numeric/double precision results instead of panicking.

Fixes: #3038

[github-actions]

@fulghum DOLT

read_tests	from_latency_median	to_latency_median	is_faster
covering_index_scan_postgres	2.43	2.43	0
groupby_scan_postgres	75.82	74.46	0
index_join_postgres	2.18	2.22	0
index_join_scan_postgres	1.58	1.58	0
index_scan_postgres	484.44	484.44	0
oltp_point_select	0.36	0.37	0
oltp_read_only	6.43	6.32	0
select_random_points	0.7	0.72	0
select_random_ranges	1.01	1.03	0
table_scan_postgres	493.24	484.44	0
types_table_scan_postgres	1213.57	1191.92	0

write_tests	from_latency_median	to_latency_median	is_faster
bulk_insert	0.001	0.001	0
oltp_delete_insert_postgres	6.67	6.55	0
oltp_insert	3.3	3.3	0
oltp_read_write	13.22	13.22	0
oltp_update_index	3.55	3.55	0
oltp_update_non_index	3.25	3.25	0
oltp_write_only	6.91	6.91	0
types_delete_insert_postgres	7.17	7.17	0

[github-actions]

	Main	PR
Total	42090	42090
Successful	18943	18981
Failures	23147	23109
Partial Successes1	5447	5461

	Main	PR
Successful	45.0059%	45.0962%
Failures	54.9941%	54.9038%

${\color{lightgreen}Progressions (38)}$

aggregates

QUERY: SELECT stddev_pop(b) FROM aggtest;

QUERY: SELECT stddev_samp(b) FROM aggtest;

QUERY: SELECT var_pop(b) FROM aggtest;

QUERY: SELECT var_samp(b) FROM aggtest;

QUERY: SELECT var_pop('inf'::float8), var_samp('inf'::float8);

QUERY: SELECT stddev_pop('inf'::float8), stddev_samp('inf'::float8);

QUERY: SELECT var_pop('nan'::float8), var_samp('nan'::float8);

QUERY: SELECT stddev_pop('nan'::float8), stddev_samp('nan'::float8);

QUERY: SELECT var_pop(1.0::float4), var_samp(2.0::float4);

QUERY: SELECT stddev_pop(3.0::float4), stddev_samp(4.0::float4);

QUERY: SELECT var_pop('inf'::float4), var_samp('inf'::float4);

QUERY: SELECT stddev_pop('inf'::float4), stddev_samp('inf'::float4);

QUERY: SELECT var_pop('nan'::float4), var_samp('nan'::float4);

QUERY: SELECT stddev_pop('nan'::float4), stddev_samp('nan'::float4);

QUERY: SELECT var_pop(1.0::numeric), var_samp(2.0::numeric);

QUERY: SELECT stddev_pop(3.0::numeric), stddev_samp(4.0::numeric);

QUERY: SELECT var_pop('nan'::numeric), var_samp('nan'::numeric);

QUERY: SELECT stddev_pop('nan'::numeric), stddev_samp('nan'::numeric);

float4

QUERY: INSERT INTO FLOAT4_TBL(f1) VALUES ('10e70'::float8);

QUERY: INSERT INTO FLOAT4_TBL(f1) VALUES ('-10e70'::float8);

QUERY: SELECT f.f1, f.f1 * '-10' AS x FROM FLOAT4_TBL f
   WHERE f.f1 > '0.0';

QUERY: SELECT f.f1, f.f1 + '-10' AS x FROM FLOAT4_TBL f
   WHERE f.f1 > '0.0';

QUERY: SELECT f.f1, f.f1 / '-10' AS x FROM FLOAT4_TBL f
   WHERE f.f1 > '0.0';

QUERY: SELECT f.f1, f.f1 - '-10' AS x FROM FLOAT4_TBL f
   WHERE f.f1 > '0.0';

numeric

QUERY: SELECT STDDEV(val) FROM num_data;

QUERY: SELECT VARIANCE(val) FROM num_data;

window

QUERY: SELECT VAR_POP(n::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
  FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);

QUERY: SELECT VAR_POP(n::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
  FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);

QUERY: SELECT VAR_POP(n::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
  FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);

QUERY: SELECT VAR_POP(n::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
  FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);

QUERY: SELECT VAR_SAMP(n::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
  FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);

QUERY: SELECT VAR_SAMP(n::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
  FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);

QUERY: SELECT VAR_SAMP(n::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
  FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);

QUERY: SELECT VAR_SAMP(n::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
  FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);

QUERY: SELECT VARIANCE(n::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
  FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);

QUERY: SELECT VARIANCE(n::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
  FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);

QUERY: SELECT VARIANCE(n::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
  FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);

QUERY: SELECT VARIANCE(n::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
  FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);

Footnotes

1. These are tests that we're marking as Successful, however they do not match the expected output in some way. This is due to small differences, such as different wording on the error messages, or the column names being incorrect while the data itself is correct. ↩

[itoqa]

Commit: ca32904: 19 test cases ran, 1 failed ❌, 18 passed ✅.

Summary

Coverage spans aggregate variance and standard-deviation behavior across numeric and floating-point inputs, grouped and moving-window calculations, null and empty-frame handling, duplicate ordering values, concurrent startup, precision limits, and recovery after extreme values. The broad results are healthy for ordinary use and many boundary conditions, but an extreme floating-point input exposes an incorrect result.

Not safe to merge yet — the PR introduces a medium-severity correctness defect in grouped and window aggregates for extreme real-valued inputs, where finite variance can become NaN. Ordinary and double-precision cases remain healthy, but this is a direct, user-visible data-result error in the changed functionality rather than a test-environment issue.

Tests run by Ito

Result	Severity	Type	Description
❌		Float	Running variance queries with an extreme REAL value returned NaN for the grouped and window population variance. The expected result is a finite value, with zero for the identical extreme values used by this check.
✅	—	Aggregate	A fresh local server registered all six variance and standard-deviation names. Grouped queries returned results for every test group without an undefined-function error or startup failure.
✅	—	Aggregate	Integer and numeric columns accepted the variance queries and returned numeric results. The checked values for 10 and 20 were correct: population variance 25 and sample standard deviation 7.07106781186547524.
✅	—	Aggregate	Real and double precision values are accepted by the variance functions, and the results are returned as double precision numbers.
✅	—	Aggregate	Four isolated workers started at the same time and all finished successfully. The variance functions stayed available, with no duplicate-registration errors or inconsistent catalog state.
✅	—	Aggregate	The aggregate functions keep the expected result type in grouped queries and window queries. Integer and decimal inputs return numeric values, while real and double inputs return double-precision values.
✅	—	Float	Grouped and window queries returned the expected population and sample variance and standard deviation values for floating-point data.
✅	—	Float	Grouped and window queries returned finite results for large, nearly equal values. The clamped partition returned zero variance and zero standard deviation, with no negative or NaN results.
✅	—	Null	Grouped calculations handled empty, one-value, and two-value groups correctly. Empty groups returned NULL, population results returned zero for one value, and sample results stayed NULL until two values were present.
✅	—	Null	Grouped and full-partition window calculations ignored empty values consistently. Empty partitions stayed NULL, one real value followed the population and sample rules, and two real values produced the expected results.
✅	—	Null	Sample variance and standard deviation stayed empty when only one real value was present, even when NULL rows were added before or after it. With two real values, padded and unpadded results matched.
✅	—	Precision	Variance and standard-deviation queries returned the expected values for ordinary numbers and numbers with a very large offset. Grouped and window results agreed, and the small spread was preserved.
✅	—	Precision	Very large decimal and BIGINT values completed without a crash or incorrect error. A normal query afterward returned variance 25 and standard deviation 5, and the database stayed available.
✅	—	Precision	A high-precision standard-deviation query returned the expected finite result. The next ordinary query returned variance 25 and standard deviation 5 in both the same and a fresh session.
✅	—	Precision	Large numeric and BIGINT values that differ by only a small amount still returned the correct positive variance in grouped and window queries.
✅	—	Window	Rows with the same ordering value receive the same cumulative variance and standard-deviation results.
✅	—	Window	Empty and backward-moving windows finish safely and return SQL NULL instead of an evaluation or division error.
✅	—	Window	Variance and standard deviation queries returned the expected values for each group, and equivalent window calculations matched those results.
✅	—	Window	Window calculations completed successfully as the frame moved between rows. No stale values or evaluation errors were observed.

Tip

Reply with @itoqa to send us feedback on this test run.

[itoqa]

Ito Diff Report — ca32904 → d54dbf0: 9 test cases ran, 9 passing ✅.

Diff Summary

Coverage focused on safe handling of floating-point and numeric values across normal boundary cases and adversarial oversized inputs. It exercised successful storage and calculation, clear rejection of invalid values, preservation of existing data, and continued connection usability after errors.

Safe to merge — the exercised behaviors completed without any PR-attributable regressions or new failures, including overflow rejection, boundary handling, data preservation, and recovery after errors. Previously passing aggregate, null-handling, precision, and window-function areas were not exercised in this run and remain a flag for later, not a merge blocker.

Tests run by Ito

Result	State	Severity	Type	Description
✅	Passing	—	Float	Large number conversions were rejected with clear errors, but the same database connection kept working. The failed insert added no row, and a later valid insert succeeded.
✅	Passing	—	Float	An oversized finite value was rejected instead of being stored. The original row stayed in place, the connection recovered, and a later valid insert succeeded.
✅	Passing	—	Float	An oversized decimal value was rejected with the expected range error, and later valid queries continued to work on the same connection.
✅	Passing	—	Float	A very large finite value was rejected instead of being stored as positive infinity, and the two existing rows stayed unchanged.
✅	Passing	—	Float	The database accepted both the largest tested positive and negative boundary values and returned finite results for each.
✅	Passing	—	Float	Inserting a huge numeric value into a real column returned a clear range error, and no row was written.
✅	Passing	—	Float	Converting a 320-digit number to double precision returned a clear out-of-range error. The same SQL connection still worked afterward, and no infinity value was returned.
✅	Passing	—	Float	Inserting an oversized number into a real column was rejected with a clear range error, and no row was inserted.
✅	Passing	—	Float	Positive and negative values at the REAL limit were stored successfully. Values beyond that limit were rejected on both sides, and the database connection stayed usable.
⏸️	Skipped	—	Aggregate	A fresh local server registered all six variance and standard-deviation names. Grouped queries returned results for every test group without an undefined-function error or startup failure.
⏸️	Skipped	—	Aggregate	Integer and numeric columns accepted the variance queries and returned numeric results. The checked values for 10 and 20 were correct: population variance 25 and sample standard deviation 7.07106781186547524.
⏸️	Skipped	—	Aggregate	Real and double precision values are accepted by the variance functions, and the results are returned as double precision numbers.
⏸️	Skipped	—	Aggregate	Four isolated workers started at the same time and all finished successfully. The variance functions stayed available, with no duplicate-registration errors or inconsistent catalog state.
⏸️	Skipped	—	Aggregate	The aggregate functions keep the expected result type in grouped queries and window queries. Integer and decimal inputs return numeric values, while real and double inputs return double-precision values.
⏸️	Skipped	—	Float	Grouped and window queries returned the expected population and sample variance and standard deviation values for floating-point data.
⏸️	Skipped	—	Float	Grouped and window queries returned finite results for large, nearly equal values. The clamped partition returned zero variance and zero standard deviation, with no negative or NaN results.
⏸️	Skipped	—	Null	Grouped calculations handled empty, one-value, and two-value groups correctly. Empty groups returned NULL, population results returned zero for one value, and sample results stayed NULL until two values were present.
⏸️	Skipped	—	Null	Grouped and full-partition window calculations ignored empty values consistently. Empty partitions stayed NULL, one real value followed the population and sample rules, and two real values produced the expected results.
⏸️	Skipped	—	Null	Sample variance and standard deviation stayed empty when only one real value was present, even when NULL rows were added before or after it. With two real values, padded and unpadded results matched.
⏸️	Skipped	—	Precision	Variance and standard-deviation queries returned the expected values for ordinary numbers and numbers with a very large offset. Grouped and window results agreed, and the small spread was preserved.
⏸️	Skipped	—	Precision	Very large decimal and BIGINT values completed without a crash or incorrect error. A normal query afterward returned variance 25 and standard deviation 5, and the database stayed available.
⏸️	Skipped	—	Precision	A high-precision standard-deviation query returned the expected finite result. The next ordinary query returned variance 25 and standard deviation 5 in both the same and a fresh session.
⏸️	Skipped	—	Precision	Large numeric and BIGINT values that differ by only a small amount still returned the correct positive variance in grouped and window queries.
⏸️	Skipped	—	Window	Rows with the same ordering value receive the same cumulative variance and standard-deviation results.
⏸️	Skipped	—	Window	Empty and backward-moving windows finish safely and return SQL NULL instead of an evaluation or division error.
⏸️	Skipped	—	Window	Variance and standard deviation queries returned the expected values for each group, and equivalent window calculations matched those results.
⏸️	Skipped	—	Window	Window calculations completed successfully as the frame moved between rows. No stale values or evaluation errors were observed.

Tip

Reply with @itoqa to send us feedback on this test run.

[Hydrocharged]

You know I like my function comments, but LGTM otherwise!

[itoqa]

Ito Diff Report — d54dbf0 → cd105ea: 14 test cases ran, 14 passing ✅.

Diff Summary

Coverage spans aggregate and window calculations across integer, exact-number, and floating-point inputs, including result types, grouping, ordering, peer handling, and frame behavior. It also exercises edge and recovery cases such as NULL-heavy or empty inputs, one-value thresholds, invalid values, overflow-style errors, and continued database usability after failures.

Safe to merge — the exercised behavior is consistently healthy, with no regressions or PR-attributable failures identified. Previously passing overflow and boundary scenarios were not rerun, but they are a flag for later rather than a merge blocker based on this run.

Tests run by Ito

Result	State	Severity	Type	Description
✅	Passing	—	Aggregate	All six variance and standard deviation names worked for integer, numeric, real, and double inputs. Integer and numeric inputs returned numeric values, while real and double inputs returned double precision values.
✅	Passing	—	Aggregate	Grouped variance queries return the expected numbers, and ordered window queries return valid results without a server error.
✅	Passing	—	Aggregate	Integer and exact-number inputs returned numeric results, while real, double, and number-to-floating-point casts returned double precision. Grouped and window queries completed without a buffer mismatch.
✅	Passing	—	Null	Grouped variance and standard deviation use the correct rules for zero, one, and two non-NULL values. All-NULL groups return NULL, one-value groups return population zero and sample NULL, and two-value groups return the expected results.
✅	Passing	—	Null	Groups with one real value returned NULL for sample variance and standard deviation, even when NULL rows surrounded that value. Groups with two real values returned the same results with or without NULL padding.
✅	Passing	—	Null	Mixed NULL and decimal values produced the same variance and standard deviation as the non-NULL values alone. A follow-up query returned 42, so the database session stayed usable.
✅	Passing	—	Null	Leading empty frames returned NULL, a frame with one value returned population zero and sample NULL, and later frames calculated normally.
✅	Passing	—	Null	Standard deviation matched the variance for unequal values, and equal values returned zero without an error.
✅	Passing	—	Null	Empty windows returned NULL, one-row windows returned population zero with no sample result, and larger windows returned the correct variance in both sort directions.
✅	Passing	—	Null	An invalid numeric query returned an error without crashing the server. A clean retry returned the correct variance and standard deviation, and a health query still worked.
✅	Passing	—	Window	Rows with the same sort-group value returned the same variance and standard-deviation results. Each later group included all rows from the groups before it.
✅	Passing	—	Window	Explicit ROWS and RANGE windows returned their expected, different values. Empty frames returned NULL, and invalid inverted-frame syntax returned a clear parser error without crashing the server.
✅	Passing	—	Window	Variance and standard-deviation windows finished successfully for integer, real, and double values. Integer results were numeric, while floating-point results were double precision, with no server panic.
✅	Passing	—	Window	ROWS frames advanced one physical row at a time, while RANGE frames gave duplicate peers the same result. Partitioned results stayed within their own group.
⏸️	Skipped	—	Float	Large number conversions were rejected with clear errors, but the same database connection kept working. The failed insert added no row, and a later valid insert succeeded.
⏸️	Skipped	—	Float	An oversized finite value was rejected instead of being stored. The original row stayed in place, the connection recovered, and a later valid insert succeeded.
⏸️	Skipped	—	Float	An oversized decimal value was rejected with the expected range error, and later valid queries continued to work on the same connection.
⏸️	Skipped	—	Float	A very large finite value was rejected instead of being stored as positive infinity, and the two existing rows stayed unchanged.
⏸️	Skipped	—	Float	The database accepted both the largest tested positive and negative boundary values and returned finite results for each.
⏸️	Skipped	—	Float	Inserting a huge numeric value into a real column returned a clear range error, and no row was written.
⏸️	Skipped	—	Float	Converting a 320-digit number to double precision returned a clear out-of-range error. The same SQL connection still worked afterward, and no infinity value was returned.
⏸️	Skipped	—	Float	Inserting an oversized number into a real column was rejected with a clear range error, and no row was inserted.
⏸️	Skipped	—	Float	Positive and negative values at the REAL limit were stored successfully. Values beyond that limit were rejected on both sides, and the database connection stayed usable.

Tip

Reply with @itoqa to send us feedback on this test run.

[itoqa]

Ito Diff Report — d54dbf0 → dde8643: 15 test cases ran, 15 passing ✅.

Diff Summary

The run covered core statistical calculations for numeric and floating-point data, including population and sample results, standard deviation, grouped and partitioned windows, and peer-aware frame behavior. It also exercised edge conditions such as nulls, single rows, duplicate values, very large or nearly equal numbers, non-finite values, and recovery after problematic queries, with healthy results throughout.

Safe to merge — the exercised calculations and edge-case behaviors showed no regressions or new failures attributable to this PR. Previously passing overflow and boundary scenarios were not rerun, so they remain a flag for later rather than a merge blocker.

Tests run by Ito

Result	State	Severity	Type	Description
✅	Passing	—	Float	Variance and standard deviation returned the expected values for both real-number and double-precision columns. The database connection also stayed usable after the queries.
✅	Passing	—	Float	Variance and standard deviation returned finite, non-negative values for both large, nearly equal float datasets. The database connection also worked after the checks.
✅	Passing	—	Float	Queries with NaN and infinity values returned the expected NaN or NULL results without crashing. A follow-up query succeeded, so the database session stayed usable.
✅	Passing	—	Float	Adding a NULL row did not change the variance or standard deviation results for the two real values. The database connection also stayed usable afterward.
✅	Passing	—	Float	REAL and DOUBLE variance and standard deviation returned matching NaN results for grouped and windowed data containing NaN and infinities. Finite values still returned the expected results, and the database connection stayed usable.
✅	Passing	—	Numeric	The database returned the expected population variance and sample variance for values 10, 20, and 30. The connection also stayed usable after the query.
✅	Passing	—	Numeric	Three identical integer values returned zero for both variance and standard deviation, and the database session remained usable.
✅	Passing	—	Numeric	One row returns zero for population variance and standard deviation, while sample results correctly return NULL. The database connection remains usable afterward.
✅	Passing	—	Numeric	Variance and standard deviation handled huge and nearly equal numbers without negative results, overflow, a server crash, or a broken session.
✅	Passing	—	Window	The two rows in each group received the same cumulative standard deviation and variance. All three duplicate peer groups passed the consistency check.
✅	Passing	—	Window	Duplicate order keys produce the same RANGE result within each partition, while ROWS advances one row at a time. Values from one partition do not affect another partition.
✅	Passing	—	Window	Bounded ROWS and RANGE calculations used only the rows in each frame, including a zero result for the first one-row frame.
✅	Passing	—	Window	Each group starts with no sample variance, then calculates variance from only its own rows.
✅	Passing	—	Window	Duplicate sort keys produced the same variance for both rows in each peer group with the default range frame. The explicit row-based frame changed the result at each row, as expected.
✅	Passing	—	Window	Each partition starts a new window frame. The first row is empty for sample statistics, and the second row returns the expected variance and standard deviation for that partition.
⏸️	Skipped	—	Float	Large number conversions were rejected with clear errors, but the same database connection kept working. The failed insert added no row, and a later valid insert succeeded.
⏸️	Skipped	—	Float	An oversized finite value was rejected instead of being stored. The original row stayed in place, the connection recovered, and a later valid insert succeeded.
⏸️	Skipped	—	Float	An oversized decimal value was rejected with the expected range error, and later valid queries continued to work on the same connection.
⏸️	Skipped	—	Float	A very large finite value was rejected instead of being stored as positive infinity, and the two existing rows stayed unchanged.
⏸️	Skipped	—	Float	The database accepted both the largest tested positive and negative boundary values and returned finite results for each.
⏸️	Skipped	—	Float	Inserting a huge numeric value into a real column returned a clear range error, and no row was written.
⏸️	Skipped	—	Float	Converting a 320-digit number to double precision returned a clear out-of-range error. The same SQL connection still worked afterward, and no infinity value was returned.
⏸️	Skipped	—	Float	Inserting an oversized number into a real column was rejected with a clear range error, and no row was inserted.
⏸️	Skipped	—	Float	Positive and negative values at the REAL limit were stored successfully. Values beyond that limit were rejected on both sides, and the database connection stayed usable.

Tip

Reply with @itoqa to send us feedback on this test run.