Observability At-Scale

[maxsmythe]

If we are load-testing a system, we should also be able to validate that the system is functioning appropriately. Observing metrics from the load-source is well-understood, observing the behaviors of Substrate are not:

• Actors can be suspended which makes prometheus-style scraping a problem (push-based may help here if we flush metrics before sleep, but that interferes with batching/sampling)
• how do we attribute resource consumption to an individual actor? Do we need to?
• Can most benchmarking metrics be satisfied via scraping atelets/API server to validate things like # of successful suspend/resume?

Logs are already persistent, are there vital metrics we should push out to logs? Do we want an easy way of re-aggregating these metrics.

Example uses:

• Validate total # of bytes written/overwritten (or FDs opened or network requests, etc.)
• Aggregate non-fatal error counts (e.g. a long-running process attempts to poll an endpoint but times out)
• Observe latency of requests as seen by actor
• Aggregate request statistics when agent is acting as a server

[alexvanboxel]

I'm willing to help here. I'm a contributor to OpenTelemetry, and I think this could be up my alley: making sure this project is OTel first, and I see in the long-term bringing some of the discoveries we're making back into the OpenTelemetry conventions.

[MushuEE]

I don't know if we want to be scraping apiserver/atelet as leaving bandwidth for critical path might be needed in load spikes. An advantage of using the DB layer to track via Debezium or PubSub. Sudo CDC. Though that would tie the observability layer to the underlying DB solution and if that is pluggable, the observation layer would need to be equally pluggable.

[maxsmythe]

Alex Van Boxel (@alexvanboxel) Would love to work with people on figuring out a solution for this.

I definitely think there is something to scraping a local FD rather than using a live endpoint -- seems like it might make observing suspended actors easier. That seems roughly analogous to Debezium?

Grant McCloskey (@MushuEE), I'm sensitive to not wanting to hog bandwidth, but there will always be some kind of resource drain associated with monitoring. Are you thinking that push-based models would be a better fit under load spikes?