Repository navigation
[DBM-720] collect SQL Server column statistics for the query benchmarker - #25769
Conversation
Adds an opt-in `collect_column_statistics` job to the metadata job. In each database it runs a customer-installed `datadog.column_statistics` procedure, which runs as its owner (the Agent's login can't read statistics itself) and returns only counts and frequencies derived from SQL Server's statistics histograms. Results are sent as the same `column_statistics` event the Postgres integration sends from pg_stats, so the benchmarker can seed SQL Server benchmark data from them unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
✅ Dispatcher tests: passed
✅ 46 passed · nothing failed Batches · ✅ batch-01 46/46 Dispatcher finished on |
|
evalya-impact-summaryEvalya CI impact |
- Reuse execute_query, and list each setup error once so its health event and failure tracking come from the same table. - Keep each database's payload batch local instead of on the long-lived collector. - Procedure: pass names to DBCC as parameters instead of escaping quotes, carry statistics and column names in #stats, drop an unused column, and index #stats so the per-statistic update doesn't scan it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Disk usage changeCommit Uncompressed
Details
Compressed
Details
|
Column statistics rode on the metadata job, so they only ran when settings collection was enabled and never more often than it. They now run as a separate DBMAsyncJob on one connection per run, with a 60s query timeout like the Postgres collector's (the check's 10s command_timeout is too short for the procedure on a large database), and max_tables passed as a parameter. The procedure reads each histogram once, calls DBCC SHOW_STATISTICS only for variable-width columns, and averages widths over non-NULL values as pg_stats does. Setup errors also match on pyodbc's native error number, so they're recognized on non-English servers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Windows setup creates the datadog user in model, so every new database already has it and CREATE USER failed, aborting setup.sql before the test environment came up. Create it only when it's missing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6391c84a67
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
More details
The completed static review raised no reportable issue in collection, payload mapping, configuration, or async-job integration. Execution against SQL Server remains unverified.
🤖 Bits Code Review · Commit 6391c84
The procedure runs WITH EXECUTE AS OWNER. On Windows the test setup connects with Windows authentication, so the database was owned by the container's Windows login, which SQL Server can't impersonate, and every call failed with error 15517. Make sa the owner in all environments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Windows SQL Server 2019 reported a null_frac of 0 for a column that is 10% NULL, though the same build on Linux reports 0.1. The procedure only counted NULLs from the histogram's NULL step, so it now falls back to the rows the other steps don't cover when that step is missing or empty. Where the step exists, the result is unchanged. The test temporarily prints the raw histogram if null_frac is still wrong, to show what Windows 2019 returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Windows 2019 passes with the NULL count fallback, so the test no longer needs to print the histogram. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…inct Read the procedure's rows one at a time, and check its columns from the cursor description so an outdated procedure is caught even when it returns no rows. The procedure now returns is_unique, so a unique column sampled below 100% gets n_distinct -1 as in Postgres. System databases are dropped in Python; the query only checks state and access. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review from AAraKKe is dismissed. Related teams and files:
- agent-integrations
- sqlserver/datadog_checks/sqlserver/column_statistics.py
- sqlserver/tests/sql/column_statistics.sql
- sqlserver/tests/test_column_statistics.py
Column statistics run once an hour, so like the Postgres collector they now run inside the metadata job on their own interval instead of in a dedicated DBMAsyncJob thread. The metadata job now runs when either settings or column statistics are enabled, and skips settings when they're off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review from AAraKKe is dismissed. Related teams and files:
- agent-integrations
- sqlserver/datadog_checks/sqlserver/column_statistics.py
- sqlserver/datadog_checks/sqlserver/metadata.py
- sqlserver/datadog_checks/sqlserver/sqlserver.py
- sqlserver/tests/test_column_statistics.py
…fields Reviewers asked that the integration collect raw statistics with SQL Server's own names and leave derivation to the backend. The procedure now returns each column's statistics properties (sys.stats, sys.dm_db_stats_properties), its type, its density vector average length, and its histogram counts, and the Agent sends them as is, with the histogram as parallel arrays. Rows are sent table by table, each table capped below the intake's per-message size by dropping its largest histograms, and payloads are split by size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SQLOLEDB, which predates datetime2, returns it as text in its own format, so the Agent failed formatting it as a datetime. The procedure now returns it as ISO 8601 text in UTC for every driver. Windows SQL Server 2019 leaves the NULL step out of sys.dm_db_stats_histogram while sys.dm_db_stats_properties counts it, so the test compares step counts only for columns without NULLs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The metadata job now runs as often as its most frequent enabled collection and gates settings by their interval like schemas and column statistics, so with settings off, schemas keep their own cadence and missed-collection health events use the job's real period. Intervals are measured between tick starts with a tenth-of-a-period tolerance, so a collection never runs more often than configured. A database whose setup error changes recovers the earlier health event, and last_updated is converted with the server's offset in whole minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ad before an error TYPE_NAME(system_type_id) is NULL for CLR types, which share system_type_id 240, so hierarchyid, geometry and geography columns had no type_name and skipped DBCC; they now use user_type_id. DBCC SHOW_STATISTICS on statistics dropped since they were listed raised error 2767 and failed the whole database, which TRY...CATCH doesn't catch inside INSERT...EXEC, so the procedure checks they still exist first. Tables completely read before a database's collection fails are now sent rather than dropped. The schema collection interval change gets its own changelog entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The procedure now returns last_updated as ISO 8601 with the server's offset, the expression activity uses for query_start, instead of converting it to UTC with a Z, so it matches the other SQL Server payloads. The test accepts the timestamp without milliseconds, which SQL Server leaves out when they're zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review from lu-zhengda is dismissed. Related teams and files:
- database-monitoring-agent
- sqlserver/changelog.d/25769.fixed
- sqlserver/datadog_checks/sqlserver/column_statistics.py
- sqlserver/tests/compose-ha/sql/aoag_primary.sql
- sqlserver/tests/compose-windows/setup.sql
- sqlserver/tests/compose/setup.sql
- sqlserver/tests/sql/column_statistics.sql
- sqlserver/tests/test_column_statistics.py
Validation ReportAll 22 validations passed. Show details
|
Motivation
The DBM query benchmarker seeds synthetic data from these statistics so benchmarked plans match the customer's. Postgres already provides them; this adds SQL Server. Per review, the integration sends raw SQL Server statistics: ddoghq/dd-go#26427 passes them through intake (must deploy first, or the fields are dropped), and ddoghq/dd-source#144127 derives the benchmarker's fields. Customer setup docs for the procedure follow in a separate docs PR.
What does this PR do?
Adds opt-in SQL Server column statistics collection (
collect_column_statistics, off by default, requiresdbm: trueand SQL Server 2017+).PostgresColumnStatisticsCollectorinPostgresMetadata, on its own interval (hourly by default). The metadata job now runs when settings or column statistics are enabled, and skips settings when they're off.get_databases(), callingEXEC datadog.column_statistics @max_tables = ?(default 500 tables per database) with a 60s query timeout, matching the Postgres collector.sys.dm_db_stats_histogramreturns no rows. So, like Postgres'sdatadog.column_statistics()function, customers install a procedure that runsWITH EXECUTE AS OWNERand returns only statistics, never column values.stats_name,auto_created,user_created,type_name,max_length,is_nullable,is_unique,last_updated(UTC),rows,rows_sampled,steps,modification_counter,average_length(density vector), andhistogramwithsys.dm_db_stats_histogram'sequal_rows,range_rows,distinct_range_rowsandaverage_range_rowsper step (neverrange_high_key). The backend derives what it needs.dbm-column-statisticstrack withddsource: sqlserver, one entry per table. Tables are sent as they're read; a table over 900 KB drops its largest histograms, and payloads are split at 4 MB.EXECUTE, with recovery once fixed.datadog_test_column_statsdatabase with deterministic data and the procedure (sharedtests/sql/column_statistics.sql) to the Linux, Windows and HA environments.Checklist before requesting review