GitOps Incident Management

[bricef]

How do we manage an incident using GitOps? How do we apply the principles to manage an incident. What happens with changes that have to occur out-of bands? What about resolving issues with the GitOps machinery itself.

Let's have a discussion to capture some of these practices that may be of use for the community.

[lloydchang]

GitOps and Incident Management are two different subject matters.

These two subject matters shouldn't be conflated at all.

---

GitOps isn't designed for Incident Management at all.

For example:

GitOps is not concerned with processes of human decision making

---

In stark contrast, Incident Management is concerned with processes of human decision making.

For example:

Roles
Incidents within a structured organization are normally dealt with by either an incident response team (IRT), or an incident management team (IMT). These are often designated beforehand or during the event, and are placed in control of the organization whilst the incident is dealt with, to restore normal functions.
https://en.wikipedia.org/wiki/Incident_management

Incidents notify the people who need to know what is happening; they also help organizations automate, manage, and track information and communications related to outages and similar problems that can impact infrastructure.
https://support.atlassian.com/opsgenie/docs/what-is-an-incident

During an Incident
Information and processes during a major incident.
> During an Incident - Information on what to do during an incident and how to constructively contribute.
> Security Incident Response - Security incidents are handled differently than normal operational incidents.
https://response.pagerduty.com/#during-an-incident

---

Thank you @bricef @scottrigby @lotharschulz @roberthstrand

[lloydchang]

@bricef asked:

How do we manage an incident using GitOps?

We don't.

My reason is explained in the above at https://github.com/gitops-working-group/gitops-working-group/discussions/128#discussioncomment-633177

Thanks @bricef

[scottrigby]

See open-gitops/documents#40