Skip to content

Stop telemetry queries from generating telemetry - #4769

Open
johnwhumphreys wants to merge 1 commit into
meta-pytorch:mainfrom
johnwhumphreys:export-D118294147
Open

Stop telemetry queries from generating telemetry#4769
johnwhumphreys wants to merge 1 commit into
meta-pytorch:mainfrom
johnwhumphreys:export-D118294147

Conversation

@johnwhumphreys

Copy link
Copy Markdown
Contributor

Summary:
Telemetry queries call TelemetryActor.scan, so they currently write into the tables they are reading. A real job on master showed each table scan added 1 messages row, 3 message_status_events rows, 2 spans rows, 4 query/scanner events rows, and 3 mailbox.posts. Four repeated queries returned scan message counts [0, 1, 2, 3] and mailbox-post totals [83, 86, 89, 92].

Mark TelemetryActor endpoints as non-recording. The same class flag disables caller-side endpoint instrumentation and receiver-side actor instrumentation, while SUPPRESS_TELEMETRY follows requests and replies. Suppressed envelopes also skip mailbox metrics and tracing. Exclude monarch_distributed_telemetry events from all tracing sinks. Application telemetry remains enabled. Remove the obsolete dashboard caveat.

The Admin /v1/query proxy uses the fixed path. /v1/tree still generates topology traffic: 3 calls changed Resolve messages from 328 to 447. Track that separately in T286999638.

Differential Revision: D118294147

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 1, 2026
@meta-codesync

meta-codesync Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

@johnwhumphreys has exported this pull request. If you are a Meta employee, you can view the originating Diff in D118294147.

Summary:
Telemetry queries call `TelemetryActor.scan`, so they currently write into the tables they are reading. A real job on master showed each table scan added 1 `messages` row, 3 `message_status_events` rows, 2 `spans` rows, 4 query/scanner `events` rows, and 3 `mailbox.posts`. Four repeated queries returned scan message counts `[0, 1, 2, 3]` and mailbox-post totals `[83, 86, 89, 92]`.

Mark `TelemetryActor` endpoints as non-recording. The same class flag disables caller-side endpoint instrumentation and receiver-side actor instrumentation, while `SUPPRESS_TELEMETRY` follows requests and replies. Suppressed envelopes also skip mailbox metrics and tracing. Exclude `monarch_distributed_telemetry` events from all tracing sinks. Application telemetry remains enabled. Remove the obsolete dashboard caveat.

The Admin `/v1/query` proxy uses the fixed path. `/v1/tree` still generates topology traffic: 3 calls changed `Resolve` messages from 328 to 447. Track that separately in T286999638.

Differential Revision: D118294147
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot. meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant