gha-runner-scale-set-0.13.1 installation fails with RunnerScaleSetNotFoundException

[erichorwath]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.13.1

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

Install 0.12.1, then upgrade to 0.13.1

Describe the bug

After upgrade, listeners are not getting ready:

Application returned an error: createSession failed: failed to create session: failed to do the session request: actions error: StatusCode 404, ActivityID "xxx" Github.Actions.Runtime.WebApi.RunnerScaleSetNotFoundException, Github.Actions.Runtime.WebApi: No runner scale set found with identifier 22.

Describe the expected behavior

listener get ready and helm upgrade succeeds.

Additional Context

K8s 1.32
GHES 3.16

Controller Logs

.

Runner Pod Logs

.

[nikola-jokic]

Hey @erichorwath,

What are the exact steps you took to upgrade?

[bavugisainath]

We are experiencing the same issue on our end. We recently created a new EKS cluster v1.35 and installed the ARC controller version 0.13.1, then deployed the scaleset v0.13.1.

After deploying the listeners, we noticed that the pods are behaving differently compared to version 0.12.1 (which we previously used). One of the listener pods continuously retries and appears to be stuck in a loop, while other listeners on the same node are functioning correctly.

The behavior seems random, as only specific listener pods are affected.

Could you please help us investigate this issue?

K8s 1.35
GHES 3.17.3
Karpenter 1.9

[jacisko]

We've done some testing with @erichorwath and performed the following steps:

gha-runner-scale-set-controller-0.12.1 chart
helm install acr . --values values.yaml

gha-runner-scale-set-0.12.1 chart
helm install runner-scale-set . --values values.yaml

gha-runner-scale-set-controller-0.13.1 chart
helm upgrade acr . --values values.yaml

gha-runner-scale-set-0.13.1 chart
helm upgrade runner-scale-set . --values values.yaml

This resulted in the below state where 1 listener pod is being recreated in a loop having error in logs:

Error:

The current runner scale set after upgrade contains runner scale set id 118, but the failing listener pod keeps referencing the old runner scale set id 117 as seen in the error:

The values files are default coming with the default charts, only thing changed was min runners changed from 0 to 1 in gha-runner-scale-set values file.

[mrbnd]

It looks like, GHA runner after update needs reregistation. What helped it my case. I fully reinstall gha-runner-scale-set helm chart(remove and install from scratch with the same values.yaml).

p.s. just restart pods didn't help

[jacisko]

Dear @nikola-jokic , is the scenario clear enough to be debugged? In case of questions let us know.
Hello @mrbnd - such an approach is not feasible in production environments.