Support custom agent download endpoints and auto-track latest release

[Copilot]

The node install script hard-coded the agent release tag (v0.0.10) and GitHub as the only download host, making air-gapped deployments and picking up new releases painful.

Install script (internal/provision/assets/unbounded-agent-install.sh)

• Default is now GitHub's releases/latest/download/ redirect, so new releases are picked up automatically without touching the script or docs.
• New AGENT_BASE_URL env var (default https://github.com/Azure/unbounded/releases) for self-hosted / mirrored release layouts.
• AGENT_VERSION still pins a specific tag; AGENT_URL remains the full-URL escape hatch.

kubectl unbounded machine manual-bootstrap

• New --agent-version, --agent-base-url, --agent-url flags. Values are POSIX-single-quoted and exported immediately before the embedded install script in both the script and cloud-init variants, so they propagate into the install-script subshell / runcmd. --agent-url works with both variants.

# Pin to a specific release:
kubectl unbounded machine manual-bootstrap my-node --site mysite --agent-version v0.0.10

# Point at an internal mirror (must mirror the GitHub releases layout):
kubectl unbounded machine manual-bootstrap my-node --site mysite \
  --agent-base-url https://releases.internal.example.com/unbounded-kube

Docs & quickstart

• Added a "Customizing the agent download" section to docs/content/guides/agent.md covering the flags/env vars and air-gapped usage.
• Extended the hack/scripts/aks-quickstart.sh "Add a Remote Node" help output to mention the three new override flags. The hack quickstart scripts (aks-quickstart.sh, proxmox-ssh-quickstart-vms.sh) don't hardcode agent URLs/versions — they print kubectl unbounded machine manual-bootstrap for the user to run and now inherit the "latest by default" behavior automatically. The .github/workflows/aks-ssh-quickstart.yaml workflow already uses --agent-url file://… and continues to work unchanged.

Merge notes

Merged origin/main (Quickstart Remote SSH Workflow, #55) and reconciled its overlapping --agent-url implementation: kept the unified InstallEnv []string template field that exports all three overrides, and dropped main's narrower AgentURLLiteral path plus the --agent-url+cloud-init rejection (cloud-init correctly exports the override in runcmd). Main's newMachineManualBootstrapCommand factory and its TestManualBootstrapHandler_Execute_WithAgentURL test are preserved.

A second merge from origin/main pulled in the unbounded-kube → unbounded repo rename; conflicts in internal/provision/assets/unbounded-agent-install.sh and internal/provision/script_test.go were resolved by keeping this PR's override logic / "latest by default" behavior and updating the default AGENT_BASE_URL (and the --agent-base-url flag help text) to https://github.com/Azure/unbounded/releases.

Out of scope

The agent's own rootfs downloads (kubelet / containerd / runc / CNI / crictl) still use hard-coded github.com and dl.k8s.io URLs in cmd/agent/internal/phases/rootfs/{cri,cni,kube}.go. Making those configurable requires extending the goal-state / AgentConfig schema and is best handled as a follow-up.

[Jason Wilder (jwilder)]

Copilot Resolve the conflicts and all check the getting started script in hack dir to make sure it is updated too.

[Philip Lombardi (plombardi89)]

Does this handle the machine register path as well cleanly?

[Jason Wilder (jwilder)]

Does this handle the machine register path as well cleanly?

Copilot check how the kubectl unbounded machine register flow will work.

[Copilot]

Does this handle the machine register path as well cleanly?

Copilot check how the kubectl unbounded machine register flow will work.

Checked the register path. Summary:

kubectl unbounded machine register doesn't run the install script directly — it creates a Machine CRD, then the machina controller SSH's in and executes provision.UnboundedAgentInstallScript() with UNBOUNDED_AGENT_CONFIG_FILE=... sudo -E bash /tmp/machina-agent-install.sh (cmd/machina/machina/controller/machine_controller.go:525, 621-625).

Because that's the same shared install script, the default behavior improves for free: register no longer pins v0.0.10 and now tracks GitHub's latest release automatically. No changes required on that path for the default case.

What register does not get from this PR is user-facing overrides. The controller's SSH exec builds a fixed command and doesn't forward AGENT_VERSION/AGENT_URL/AGENT_BASE_URL, and Machine.Spec.Agent (AgentSpec in api/machina/v1alpha3/machine_types.go:267) only has an Image field for the nspawn OCI image — nothing for the agent-binary download. So "pin to v0.0.10" or "use an internal mirror" via register would require:

1. Adding Version / URL / BaseURL fields to AgentSpec (CRD + deepcopy regeneration).
2. Plumbing them through MachineReconciler so the ssh exec command prefixes AGENT_VERSION=… AGENT_URL=… AGENT_BASE_URL=… before sudo -E bash.
3. Adding matching --agent-version / --agent-url / --agent-base-url flags on kubectl unbounded machine register.

That's a non-trivial schema/controller change and feels like it belongs in the same follow-up as the agent's own rootfs-download overrides (noted under "Out of scope" in the PR description). Happy to do it in this PR if you'd prefer — just let me know.

[Jason Wilder (jwilder)]

Copilot Create a new issue to track the recommendations for a follow-up PR for the AgentSpec and other issues identified that are out of scope. Make sure to include all relevant discovered information so it can be implemented correctly in a follow-up PR. It should account for the machine register case as well as overriding the kubelet/cri/etc.. binary locations and versions. Do not implement the PR yet.

[Copilot]

Copilot Create a new issue to track the recommendations for a follow-up PR for the AgentSpec and other issues identified that are out of scope. Make sure to include all relevant disc...

My toolset here doesn't include an issue-creation API, so I can't file the issue directly. Full draft below - copy/paste into a new issue as-is.

---

Title: Make all agent/node download endpoints configurable (agent binary via machine register, plus kubelet/containerd/runc/CNI/crictl)

Body:

Follow-up to #. That PR made the unbounded-agent binary download URL configurable for kubectl unbounded machine manual-bootstrap via AGENT_VERSION / AGENT_BASE_URL / AGENT_URL env vars + matching flags, and switched the default from a pinned v0.0.10 tag to GitHub's releases/latest/download/ redirect.

Two gaps remain that this issue tracks:

1. machine register / machina controller path

kubectl unbounded machine register creates a Machine CRD; the machina controller then SSH's to the host and executes the shared install script. The default "latest" behavior already works there for free, but there is no way to pin a version or point at a mirror through register.

Relevant code:

• cmd/kubectl-unbounded/app/machine_register.go - machineRegisterHandler + flags (no agent-download flags today).
• cmd/machina/machina/controller/machine_controller.go:521-629 - SSH copy of provision.UnboundedAgentInstallScript() to /tmp/machina-agent-install.sh and exec of UNBOUNDED_AGENT_CONFIG_FILE=%q sudo -E bash %s. No env forwarding for AGENT_*.
• api/machina/v1alpha3/machine_types.go:267-273 - AgentSpec currently only has Image string (nspawn OCI image); no fields for agent-binary download.
• internal/provision/assets/unbounded-agent-install.sh - already honors AGENT_VERSION / AGENT_BASE_URL / AGENT_URL.

Proposed work:

1. Extend AgentSpec with optional Version, BaseURL, URL fields (kubebuilder +optional); regenerate zz_generated.deepcopy.go and the CRD YAML via controller-gen crd paths="./api/..." output:crd:artifacts:config=deploy/machina/crd (CI verifies generated-file drift).
2. In MachineReconciler.runAgentInstallScript (the code around machine_controller.go:621), prepend POSIX-quoted AGENT_VERSION=... AGENT_BASE_URL=... AGENT_URL=... to the command based on machine.Spec.Agent. Reuse the single-quoting helper already added in cmd/kubectl-unbounded/app/machine_manual_bootstrap.go (factor it into internal/provision or a shared helper).
3. Add --agent-version, --agent-base-url, --agent-url flags to kubectl unbounded machine register that populate Machine.Spec.Agent.{Version,BaseURL,URL}.
4. Tests: unit tests for the new flag plumbing in machine_register_test.go and controller tests that assert the env prefix appears in the exec command.
5. Docs: extend the "Customizing the agent download" section in docs/content/guides/agent.md with the register variants and a Machine YAML example.

2. Agent-internal rootfs downloads (kubelet / containerd / runc / CNI / crictl)

The agent itself then downloads five more binaries into the nspawn rootfs from hard-coded hosts:

• cmd/agent/internal/phases/rootfs/kube.go:28 kubernetesBinaryURLTemplate = "https://dl.k8s.io/v%s/bin/linux/%s/%s"
• cmd/agent/internal/phases/rootfs/kube.go:33 kubernetesChecksumURLTemplate = "https://dl.k8s.io/v%s/bin/linux/%s/%s.sha256"
• cmd/agent/internal/phases/rootfs/kube.go:37 criToolsCrictlURLTemplate = "https://github.com/kubernetes-sigs/cri-tools/releases/download/v%s/crictl-v%s-%s-%s.tar.gz"
• cmd/agent/internal/phases/rootfs/cri.go:22 containerdURLTemplate = "https://github.com/containerd/containerd/releases/download/v%s/containerd-%s-linux-%s.tar.gz"
• cmd/agent/internal/phases/rootfs/cri.go:29 runcURLTemplate = "https://github.com/opencontainers/runc/releases/download/v%s/runc.%s"
• cmd/agent/internal/phases/rootfs/cni.go:25 cniPluginsURLTemplate = "https://github.com/containernetworking/plugins/releases/download/v%s/cni-plugins-linux-%s-v%s.tgz"

None of these are overridable, so air-gapped / mirrored deployments currently can't use machina at all past the agent itself.

Proposed work:

1. Extend the goal-state / AgentConfig schema (see internal/provision/agent_config.go and the agent-side config in cmd/agent/internal/cmd/config.go) with a Downloads block, e.g.:

   type Downloads struct {
       Kubernetes *DownloadSource `json:"kubernetes,omitempty"` // overrides dl.k8s.io
       Containerd *DownloadSource `json:"containerd,omitempty"`
       Runc       *DownloadSource `json:"runc,omitempty"`
       CNI        *DownloadSource `json:"cni,omitempty"`
       Crictl     *DownloadSource `json:"crictl,omitempty"`
   }
   type DownloadSource struct {
       BaseURL string `json:"baseURL,omitempty"` // replaces the host + path prefix; keeps version/arch substitution
       URL     string `json:"url,omitempty"`     // full URL escape hatch (version/arch still substituted via %s)
       Version string `json:"version,omitempty"` // override the version that would otherwise be derived from Cluster.Version
   }

   (Preserve existing crictl minor-alignment behavior: Kubernetes vX.Y.Z -> cri-tools vX.Y.0, see cmd/agent/internal/phases/rootfs/kube.go:220-228.)
2. Thread the config through each rootfs/*.go downloader so the *URLTemplate constants become defaults that can be overridden per-binary.
3. Mirror the new fields on the Machine CRD (probably under Spec.Agent or a new Spec.Downloads), regenerate CRDs/deepcopy, and plumb them into BuildAgentConfig in internal/provision/agent_config.go.
4. Add corresponding --kubelet-base-url, --containerd-base-url, etc. flags to kubectl unbounded machine register and manual-bootstrap (or a single --download-override key=value repeatable flag).
5. Tests in cmd/agent/internal/phases/rootfs/{cri,cni,kube}_test.go covering the override paths; integration coverage via the existing smoke test (hack/smoke-metalman.py) pointed at a local mirror.
6. Docs update: expand the "Customizing the agent download" guide with an air-gapped end-to-end example covering all six artifacts.

Acceptance criteria

• Operator can register a node against an internal mirror for all six downloads (agent + kubelet + containerd + runc + CNI + crictl) without touching source.
• Operator can pin each artifact to a specific version independently of cluster Kubernetes version (with the current minor-alignment default preserved for crictl).
• manual-bootstrap, register, and the machina controller path all expose the same override surface.
• Existing default behavior (GitHub latest for agent, dl.k8s.io for kube binaries, upstream GitHub releases for containerd/runc/CNI/crictl) is unchanged when no overrides are set.
• CRDs / deepcopy regenerated and CI "Verify Generated Files" passes.

Non-goals

• Changing the actual artifacts (still expected to be upstream-compatible tarballs with the same layout).
• Mirroring layout discovery - users configure base URLs explicitly.
• Signature / checksum policy changes beyond what dl.k8s.io already provides.

[Israel Miller (imiller31)]

Copilot validate that no changes to the AKS SSH Quickstart workflow are needed due to your changes

[Copilot]

Copilot validate that no changes to the AKS SSH Quickstart workflow are needed due to your changes

Validated - no changes needed to .github/workflows/aks-ssh-quickstart.yaml.

The workflow builds the agent locally, scps the tarball to each VM, then invokes kubectl unbounded machine manual-bootstrap ... --agent-url file:///tmp/unbounded-agent-linux-amd64.tar.gz (lines 451-455 and 466-470). That --agent-url flag is preserved by this PR (cmd/kubectl-unbounded/app/machine_manual_bootstrap.go:456) and still takes precedence over --agent-version / --agent-base-url, so:

• The workflow's existing flag/value works unchanged.
• The new default (GitHub releases/latest/download/) never kicks in for this workflow because --agent-url is always set, so there's no risk of accidentally pulling a random upstream release in CI.
• TestManualBootstrapHandler_Execute_WithAgentURL in machine_manual_bootstrap_test.go exercises the same flag end-to-end.

[Israel Miller (imiller31)]

Copilot resolve the merge conflicts in this PR

[Copilot]

Copilot resolve the merge conflicts in this PR

Merge conflicts resolved in e5e7cef.

Conflicts were in internal/provision/assets/unbounded-agent-install.sh and internal/provision/script_test.go, caused by the unbounded-kube → unbounded repo rename on main. Kept this PR's AGENT_VERSION / AGENT_BASE_URL / AGENT_URL override logic and "latest by default" behavior, and updated the default base URL to https://github.com/Azure/unbounded/releases (also in the --agent-base-url flag help in machine_manual_bootstrap.go). go build ./... and go test ./internal/provision/... ./cmd/kubectl-unbounded/app/... -run ManualBootstrap pass.