feat(runner): add live-log streaming metrics - #7947
Conversation
Greptile SummaryThis PR adds Prometheus observability to the agent's live-log streaming layer — pure additive instrumentation with no behaviour change. Counters and the duration histogram are recorded inline at each event site; session-count and replay-bytes gauges use a pull-based custom collector that reads from the session managers at scrape time, keeping the gauges permanently in step with the managers without any explicit increment/decrement bookkeeping.
Confidence Score: 5/5
Important Files Changed
Sequence DiagramsequenceDiagram
participant Manager as SessionManager
participant Metrics as liveLogCollector
participant Prom as Prometheus
Note over Manager,Prom: Session creation
Manager->>Manager: attach(req) — lock held
Manager->>Manager: removeSessionLocked(old, "replaced")
Manager->>Manager: "sessions[key] = newSession"
Manager->>Prom: liveLogSessionsCreatedTotal.Inc()
Note over Manager,Prom: Source lifetime
Manager->>Manager: runSource(ctx, key, session)
Manager->>Prom: liveLogSourceDurationSeconds.Observe()
Manager->>Manager: scheduleExpiration(key, session)
Note over Manager,Prom: Scrape time (gauges)
Prom->>Metrics: Collect()
Metrics->>Metrics: totalsByKind() — snapshot sources (collector lock)
Metrics->>Manager: liveLogStats() — read sessions (manager lock)
Manager-->>Metrics: "{active, done, subscribers, replayBytes}"
Metrics-->>Prom: MustNewConstMetric × 4 per kind
Note over Manager,Prom: Context ends
Manager->>Metrics: liveLogMetrics.remove(manager)
Reviews (2): Last reviewed commit: "test(runner): trim the live-log metrics ..." | Re-trigger Greptile |
| "github.com/kubeshop/testkube/pkg/repository/channels" | ||
| ) | ||
|
|
||
| func TestLiveLogMetricsMoveOnAttachAndPublish(t *testing.T) { |
There was a problem hiding this comment.
Do you think this metrics_test should be testing the implementation of the notification stream session manager? It feels like the metrics just defines the metrics and we should have the tests living alongside other tests for the stream manager.
There was a problem hiding this comment.
Makes sense, I'll port it so the metrics are tested through an actual flow and test it in testworklows_test.go
5bb0132 to
30f8c03
Compare
✅ Testkube GitHub IntegrationReview based on commit All tests and quality gates passed.
7 workflows executed
|
2 participants
How
Instruments the agent's live-log notification stream (the session manager, replay buffer, and resume path in
pkg/controlplaneclient) with Prometheus metrics so we can build dashboards and alerts for live-log streaming. Pure observability, additive, no behaviour change.Metrics are registered on the default registry, which the agent already serves at
/metrics, and are labelled by streamkind(workflow, parallel, service).Gauges are read at scrape time from the live session managers through one collector, so they always equal the state the managers hold and no code path has to keep them in step:
testkube_live_log_sessions(kind,state=active|done) — live and draining session counts.testkube_live_log_replay_bytes(kind) — total buffered replay memory, the number to graph against the buffer budget.testkube_live_log_subscribers(kind) — concurrent viewers.Counters and the histogram record events where they happen:
testkube_live_log_sessions_created_total(kind).testkube_live_log_sessions_evicted_total(kind,reason=ttl|errored|replaced) — every removal from the manager goes through one function, so each eviction is counted once with its reason.testkube_live_log_resume_total(kind,result=available|unavailable) — how often a reconnect actually resumed versus fell back to a full reset. This is the signal that the resume path is working in production.testkube_live_log_source_duration_seconds(kind,result=ok|error) — buckets span 1 s to 2 h, because a source lives as long as the execution it follows.The replay buffer limits become fields on the manager so the tests can set them. The tests live next to the other session manager tests in
testworkflows_test.goand assert the manager's stats snapshot directly; the collector has one test against its exposition text.