Skip to content

[DBM-720] collect SQL Server column statistics for the query benchmarker - #25769

Merged
madisonmussari merged 18 commits into
masterfrom
madisonmussari/sqlserver-column-statistics
Oct 28, 2026
Merged

madisonmussari merged 18 commits into
masterfrom
madisonmussari/sqlserver-column-statistics

Conversation

@madisonmussari

@madisonmussari madisonmussari commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

The DBM query benchmarker seeds synthetic data from these statistics so benchmarked plans match the customer's. Postgres already provides them; this adds SQL Server. Per review, the integration sends raw SQL Server statistics: ddoghq/dd-go#26427 passes them through intake (must deploy first, or the fields are dropped), and ddoghq/dd-source#144127 derives the benchmarker's fields. Customer setup docs for the procedure follow in a separate docs PR.

What does this PR do?

Adds opt-in SQL Server column statistics collection (collect_column_statistics, off by default, requires dbm: true and SQL Server 2017+).

  • Runs inside the existing metadata job, like PostgresColumnStatisticsCollector in PostgresMetadata, on its own interval (hourly by default). The metadata job now runs when settings or column statistics are enabled, and skips settings when they're off.
  • Collects from the online, accessible user databases among get_databases(), calling EXEC datadog.column_statistics @max_tables = ? (default 500 tables per database) with a 60s query timeout, matching the Postgres collector.
  • The Agent's login can't read statistics: with the standard DBM grants, sys.dm_db_stats_histogram returns no rows. So, like Postgres's datadog.column_statistics() function, customers install a procedure that runs WITH EXECUTE AS OWNER and returns only statistics, never column values.
  • Sends each column's statistics as SQL Server reports them, under SQL Server's names: stats_name, auto_created, user_created, type_name, max_length, is_nullable, is_unique, last_updated (UTC), rows, rows_sampled, steps, modification_counter, average_length (density vector), and histogram with sys.dm_db_stats_histogram's equal_rows, range_rows, distinct_range_rows and average_range_rows per step (never range_high_key). The backend derives what it needs.
  • Sent on the existing dbm-column-statistics track with ddsource: sqlserver, one entry per table. Tables are sent as they're read; a table over 900 KB drops its largest histograms, and payloads are split at 4 MB.
  • Health events when the procedure is missing, outdated, or the Agent lacks EXECUTE, with recovery once fixed.
  • Test setup adds a datadog_test_column_stats database with deterministic data and the procedure (shared tests/sql/column_statistics.sql) to the Linux, Windows and HA environments.

Checklist before requesting review

  • I self-reviewed the full diff, and this description matches the final change
  • Tests cover the change, or this description explains why none are needed
  • Automated review comments, including Codex review, were addressed or answered
  • I, the human author, understand this change, can explain it to reviewers, and believe it meets the repository standards

Adds an opt-in `collect_column_statistics` job to the metadata job. In each
database it runs a customer-installed `datadog.column_statistics` procedure,
which runs as its owner (the Agent's login can't read statistics itself) and
returns only counts and frequencies derived from SQL Server's statistics
histograms. Results are sent as the same `column_statistics` event the
Postgres integration sends from pg_stats, so the benchmarker can seed
SQL Server benchmark data from them unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dd-octo-sts

dd-octo-sts Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dispatcher tests: passed

Dispatcher beta: informational only
Existing CI remains the merge signal.

  46/46 jobs

✅ 46 passed · nothing failed

Batches · ✅ batch-01 46/46

Dispatcher finished on 4d85af4 — GitHub Run · Dispatcher Logs.

@datadog-official

datadog-official Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Tests  Pipelines  Code Coverage

⚠️ Warnings

Your PR has warnings. Please review the issues below.

❄️ 1 New flaky test detected

test_collects_column_statistics from test_column_statistics.py
assert None
 &#43;  where None = &lt;function fullmatch at 0x7ff47043fe20&gt;(&#39;\ &#39;2026-10-28T17:26:15Z&#39;)
 &#43;    where &lt;function fullmatch at 0x7ff47043fe20&gt; = re.fullmatch

View in Flaky Test Management

ℹ️ Info

No other issues found (see more)

🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 97.08%
• Overall Coverage: 91.66% (+2.10%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 283258b | Docs | View more details | Give us feedback!

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

evalya-impact-summary

Evalya CI impact

Impact analysis: RUN-ALL — every test task will run
Trigger:         empty diff (default branch, scheduled run, or shallow-clone fallback)
Test tasks:      0 (all selected)
Publish tasks:   4 (always emitted)
Changed files:   0

Full impact report
Learn more about CI impact filtering

- Reuse execute_query, and list each setup error once so its health event
  and failure tracking come from the same table.
- Keep each database's payload batch local instead of on the long-lived
  collector.
- Procedure: pass names to DBCC as parameters instead of escaping quotes,
  carry statistics and column names in #stats, drop an unused column, and
  index #stats so the per-statistic update doesn't scan it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dd-octo-sts

dd-octo-sts Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Disk usage change

Commit 283258b compared against 9c3b5d6.

Uncompressed

linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
🔺 +19.7 KiB 🔺 +19.7 KiB 🔺 +19.7 KiB 🔺 +19.7 KiB 🔺 +19.7 KiB
Details
Name Version Type linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
sqlserver 23.6.0 Integration 🔺 +19.7 KiB 🔺 +19.7 KiB 🔺 +19.7 KiB 🔺 +19.7 KiB 🔺 +19.7 KiB

Compressed

linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
🔺 +6.04 KiB 🔺 +6.04 KiB 🔺 +6.04 KiB 🔺 +6.04 KiB 🔺 +6.04 KiB
Details
Name Version Type linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
sqlserver 23.6.0 Integration 🔺 +6.04 KiB 🔺 +6.04 KiB 🔺 +6.04 KiB 🔺 +6.04 KiB 🔺 +6.04 KiB

madisonmussari and others added 2 commits October 26, 2026 09:10
Column statistics rode on the metadata job, so they only ran when settings
collection was enabled and never more often than it. They now run as a
separate DBMAsyncJob on one connection per run, with a 60s query timeout
like the Postgres collector's (the check's 10s command_timeout is too short
for the procedure on a large database), and max_tables passed as a
parameter.

The procedure reads each histogram once, calls DBCC SHOW_STATISTICS only
for variable-width columns, and averages widths over non-NULL values as
pg_stats does. Setup errors also match on pyodbc's native error number, so
they're recognized on non-English servers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Windows setup creates the datadog user in model, so every new
database already has it and CREATE USER failed, aborting setup.sql
before the test environment came up. Create it only when it's missing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@madisonmussari madisonmussari changed the title Collect SQL Server column statistics for the query benchmarker [DBM-720] collect SQL Server column statistics for the query benchmarker Oct 26, 2026
@madisonmussari
madisonmussari marked this pull request as ready for review October 26, 2026 14:30
@madisonmussari
madisonmussari requested review from a team as code owners October 26, 2026 14:30
@linear-code

linear-code Bot commented Oct 26, 2026

Copy link
Copy Markdown

DBM-720

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-26T14:34:36.825335Z 6391c84 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6391c84a67

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@datadog-official datadog-official Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The completed static review raised no reportable issue in collection, payload mapping, configuration, or async-job integration. Execution against SQL Server remains unverified.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 6391c84

madisonmussari and others added 4 commits October 26, 2026 11:00
The procedure runs WITH EXECUTE AS OWNER. On Windows the test setup
connects with Windows authentication, so the database was owned by the
container's Windows login, which SQL Server can't impersonate, and every
call failed with error 15517. Make sa the owner in all environments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Windows SQL Server 2019 reported a null_frac of 0 for a column that is
10% NULL, though the same build on Linux reports 0.1. The procedure only
counted NULLs from the histogram's NULL step, so it now falls back to the
rows the other steps don't cover when that step is missing or empty.
Where the step exists, the result is unchanged.

The test temporarily prints the raw histogram if null_frac is still wrong,
to show what Windows 2019 returns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Windows 2019 passes with the NULL count fallback, so the test no longer
needs to print the histogram.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…inct

Read the procedure's rows one at a time, and check its columns from the
cursor description so an outdated procedure is caught even when it returns
no rows. The procedure now returns is_unique, so a unique column sampled
below 100% gets n_distinct -1 as in Postgres. System databases are dropped
in Python; the query only checks state and access.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@temporal-github-worker-1
temporal-github-worker-1 Bot dismissed AAraKKe’s stale review October 28, 2026 13:32

Review from AAraKKe is dismissed. Related teams and files:

  • agent-integrations
    • sqlserver/datadog_checks/sqlserver/column_statistics.py
    • sqlserver/tests/sql/column_statistics.sql
    • sqlserver/tests/test_column_statistics.py
AAraKKe
AAraKKe previously approved these changes Oct 28, 2026

@AAraKKe AAraKKe left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀🚀🚀

Column statistics run once an hour, so like the Postgres collector they now
run inside the metadata job on their own interval instead of in a dedicated
DBMAsyncJob thread. The metadata job now runs when either settings or column
statistics are enabled, and skips settings when they're off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@temporal-github-worker-1
temporal-github-worker-1 Bot dismissed AAraKKe’s stale review October 28, 2026 14:57

Review from AAraKKe is dismissed. Related teams and files:

  • agent-integrations
    • sqlserver/datadog_checks/sqlserver/column_statistics.py
    • sqlserver/datadog_checks/sqlserver/metadata.py
    • sqlserver/datadog_checks/sqlserver/sqlserver.py
    • sqlserver/tests/test_column_statistics.py
madisonmussari and others added 3 commits October 28, 2026 11:38
…fields

Reviewers asked that the integration collect raw statistics with SQL
Server's own names and leave derivation to the backend. The procedure now
returns each column's statistics properties (sys.stats,
sys.dm_db_stats_properties), its type, its density vector average length,
and its histogram counts, and the Agent sends them as is, with the
histogram as parallel arrays. Rows are sent table by table, each table
capped below the intake's per-message size by dropping its largest
histograms, and payloads are split by size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SQLOLEDB, which predates datetime2, returns it as text in its own
format, so the Agent failed formatting it as a datetime. The procedure
now returns it as ISO 8601 text in UTC for every driver. Windows SQL
Server 2019 leaves the NULL step out of sys.dm_db_stats_histogram while
sys.dm_db_stats_properties counts it, so the test compares step counts
only for columns without NULLs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The metadata job now runs as often as its most frequent enabled
collection and gates settings by their interval like schemas and column
statistics, so with settings off, schemas keep their own cadence and
missed-collection health events use the job's real period. Intervals are
measured between tick starts with a tenth-of-a-period tolerance, so a
collection never runs more often than configured. A database whose setup
error changes recovers the earlier health event, and last_updated is
converted with the server's offset in whole minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lu-zhengda
lu-zhengda previously approved these changes Oct 28, 2026
…ad before an error

TYPE_NAME(system_type_id) is NULL for CLR types, which share system_type_id
240, so hierarchyid, geometry and geography columns had no type_name and
skipped DBCC; they now use user_type_id. DBCC SHOW_STATISTICS on statistics
dropped since they were listed raised error 2767 and failed the whole
database, which TRY...CATCH doesn't catch inside INSERT...EXEC, so the
procedure checks they still exist first. Tables completely read before a
database's collection fails are now sent rather than dropped. The schema
collection interval change gets its own changelog entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The procedure now returns last_updated as ISO 8601 with the server's
offset, the expression activity uses for query_start, instead of
converting it to UTC with a Z, so it matches the other SQL Server
payloads. The test accepts the timestamp without milliseconds, which
SQL Server leaves out when they're zero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@temporal-github-worker-1
temporal-github-worker-1 Bot dismissed lu-zhengda’s stale review October 28, 2026 20:11

Review from lu-zhengda is dismissed. Related teams and files:

  • database-monitoring-agent
    • sqlserver/changelog.d/25769.fixed
    • sqlserver/datadog_checks/sqlserver/column_statistics.py
    • sqlserver/tests/compose-ha/sql/aoag_primary.sql
    • sqlserver/tests/compose-windows/setup.sql
    • sqlserver/tests/compose/setup.sql
    • sqlserver/tests/sql/column_statistics.sql
    • sqlserver/tests/test_column_statistics.py
@dd-octo-sts

dd-octo-sts Bot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Validation Report

All 22 validations passed.

Show details
Validation Description Status
agent-reqs Verify check versions match the Agent requirements file ✅
ci Validate CI configuration and code coverage settings ✅
codeowners Validate every integration has a CODEOWNERS entry ✅
config Validate default configuration files against spec.yaml ✅
dep Verify dependency pins are consistent and Agent-compatible ✅
http Validate integrations use the HTTP wrapper correctly ✅
imports Validate check imports do not use deprecated modules ✅
integration-style Validate check code style conventions ✅
jmx-metrics Validate JMX metrics definition files and config ✅
labeler Validate PR labeler config matches integration directories ✅
legacy-signature Validate no integration uses the legacy Agent check signature ✅
license-headers Validate Python files have proper license headers ✅
licenses Validate third-party license attribution list ✅
metadata Validate metadata.csv metric definitions ✅
models Validate configuration data models match spec.yaml ✅
openmetrics Validate OpenMetrics integrations disable the metric limit ✅
package Validate Python package metadata, naming, and wheel contents ✅
pr-description Validate the pull request checklist and visible description length ✅
qa-label Validate the pull request declares whether it needs QA for the next Agent release ✅
readmes Validate README files have required sections ✅
saved-views Validate saved view JSON file structure and fields ✅
version Validate version consistency between package and changelog ✅

View full run

@madisonmussari
madisonmussari added this pull request to the merge queue Oct 28, 2026
Merged via the queue into master with commit 435229c Oct 28, 2026
121 of 124 checks passed
@madisonmussari
madisonmussari deleted the madisonmussari/sqlserver-column-statistics branch October 28, 2026 20:50
@dd-octo-sts dd-octo-sts Bot added this to the 7.86.0 milestone Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants