Requestor: @kvalliyurnatt
Summary
Ignore nodes that are already cordoned (not by any of the GPU operator components) during upgrades from upgrade controller/k8s-driver-manager
Motivation
If there is a node that has already been cordoned (not by any of the GPU operator components), then that means the node is not accepting any workloads at the moment, and there is no guarantee of the state that node is in, it could be down or in an unresponsive state. So it feels like the upgrade controller/k8s-driver-manager should ignore nodes that are already cordoned and not take any action on such nodes until they are uncordoned.
Proposal
We will need a way differentiate nodes that are cordoned by the GPU operator components, for which we could add an annotation on the nodes when the upgrade controller/k8s-driver-manager cordons a node(I believe the upgrade controller already does this) and then uncordon nodes based on that annotation. Then the upgrade controller/k8s-driver-manager can ignore nodes that are in a cordoned state but don't have the annotation on them.
Requestor: @kvalliyurnatt
Summary
Ignore nodes that are already cordoned (not by any of the GPU operator components) during upgrades from upgrade controller/k8s-driver-manager
Motivation
If there is a node that has already been cordoned (not by any of the GPU operator components), then that means the node is not accepting any workloads at the moment, and there is no guarantee of the state that node is in, it could be down or in an unresponsive state. So it feels like the upgrade controller/k8s-driver-manager should ignore nodes that are already cordoned and not take any action on such nodes until they are uncordoned.
Proposal
We will need a way differentiate nodes that are cordoned by the GPU operator components, for which we could add an annotation on the nodes when the upgrade controller/k8s-driver-manager cordons a node(I believe the upgrade controller already does this) and then uncordon nodes based on that annotation. Then the upgrade controller/k8s-driver-manager can ignore nodes that are in a cordoned state but don't have the annotation on them.