Skip to content

Sync host & service check outputs to Notifications periodically - #1156

Merged
oxzi merged 1 commit into
mainfrom
check-output-syncer
Jul 30, 2026
Merged

Sync host & service check outputs to Notifications periodically#1156
oxzi merged 1 commit into
mainfrom
check-output-syncer

Conversation

@yhabteab

@yhabteab yhabteab commented Jul 21, 2026

Copy link
Copy Markdown
Member

In order to always have the latest check output of a checkable, there is now a new goroutine that periodically (every 5 minutes) syncs the check output of all checkables that have an active incident state from the icinga:{host,service}:state Redis hashes to Icinga Notifications. This is necessary because a checkable's check can, for instance, be in CRITICAL state for a long time, while its output can change with each check result. And since that won't trigger any state change events, thus, not queued to the runtime updates stream, there's no way for Icinga Notifications to know about the updated output.

icingadb-1  | 2026-07-21T13:08:45.586+0200	DEBUG	notifications	Syncing check outputs for incidents	{"host_incidents": 0, "service_incidents": 10, "last_sync": "2026-07-21T13:03:45.586+0200"}
icingadb-1  | 2026-07-21T13:13:45.650+0200	DEBUG	notifications	Syncing check outputs for incidents {"host_incidents": 0, "service_incidents": 10, "last_sync": "2026-07-21T13:08:45.645+0200"}
icingadb-1  | 2026-07-21T13:18:45.719+0200	DEBUG	notifications	Syncing check outputs for incidents {"host_incidents": 0, "service_incidents": 10, "last_sync": "2026-07-21T13:13:45.718+0200"}
icingadb-1  | 2026-07-21T13:23:45.792+0200	DEBUG	notifications	Syncing check outputs for incidents	{"host_incidents": 0, "service_incidents": 10, "last_sync": "2026-07-21T13:18:45.792+0200"}
icingadb-1  | 2026-07-21T13:28:45.869+0200	DEBUG	notifications	Syncing check outputs for incidents	{"host_incidents": 0, "service_incidents": 10, "last_sync": "2026-07-21T13:23:45.869+0200"}

resolves #1140

@yhabteab yhabteab added this to the 1.6.0 milestone Jul 21, 2026
@yhabteab
yhabteab requested a review from oxzi July 21, 2026 11:23
@cla-bot cla-bot Bot added the cla/signed label Jul 21, 2026
Comment thread pkg/notifications/notifications.go Outdated
Comment thread pkg/notifications/notifications.go Outdated
Comment thread pkg/notifications/notifications.go Outdated
@yhabteab
yhabteab force-pushed the check-output-syncer branch from f02d816 to bab0421 Compare July 29, 2026 09:43
@yhabteab
yhabteab requested a review from oxzi July 29, 2026 09:44
Comment thread pkg/notifications/notifications.go Outdated
Comment thread pkg/notifications/notifications.go Outdated
In order to always have the latest check output of a checkable, there is
now a new goroutine that periodically (every 5 minutes) syncs the check
output of all checkables that have an active incident state from the
`icinga:{host,service}:state` Redis hashes to Icinga Notifications. This
is necessary because a checkable's check can, for instance, be in
`CRITICAL` state for a long time, while its output can change with each
check result. And since that won't trigger any state change events,
thus, not queued to the runtime updates stream, there's no way for
Icinga Notifications to know about the updated output.
@yhabteab
yhabteab force-pushed the check-output-syncer branch from bab0421 to 0edfcfc Compare July 29, 2026 14:25
@yhabteab
yhabteab requested a review from oxzi July 29, 2026 14:28
@oxzi
oxzi merged commit 1f63fe3 into main Jul 30, 2026
31 checks passed
@oxzi
oxzi deleted the check-output-syncer branch July 30, 2026 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Continuously synchronize open Icinga Notification Incidents with updated check outputs

2 participants