Skip to content

feat(karpenter-gcp): update chart to upstream v0.6.0 - #318

Open
jatintalgotra-zd wants to merge 3 commits into
mainfrom
karpenter-gcp-v0.6.0
Open

feat(karpenter-gcp): update chart to upstream v0.6.0#318
jatintalgotra-zd wants to merge 3 commits into
mainfrom
karpenter-gcp-v0.6.0

Conversation

@jatintalgotra-zd

Copy link
Copy Markdown
Contributor

The chart was forked from karpenter-provider-gcp around October 2025 and tracked app version v0.0.1. Upstream is now at v0.6.0 — 11 releases ahead. This syncs it.

Why this matters

Upstream renamed the controller's LOCATION env var to CLUSTER_LOCATION and added NODE_LOCATION. The chart still set the old name, so any recent image would start with no location configured. That is the headline fix.

What changed

CRDs — the largest part of the diff. GCENodeClass grew 413 → 681 lines. Previously missing: networkConfig (private nodes, additional interfaces), shieldedInstanceConfig, confidentialInstanceType, disk IOPS/throughput, subnetRangeName, autoGPUTaint, gpuDriverVersion, KMS keys, secondary boot disks, structured imageSelectorTerms, and the kubelet configuration fields. NodePool/NodeClaim move from karpenter-core v1.7 to v1.14. The NodeOverlay CRD did not exist here at all and is added with its RBAC.

Controller config — location rename, plus IGNORE_DRA_REQUESTS, DEFAULT_NODEPOOL_TEMPLATE_NAME, DEFAULT_NODEPOOL_SERVICE_ACCOUNT, DISABLE_CONTROLLER_WARMUP. All six feature gates are now configurable; only SpotToSpotConsolidation was being passed.

Templates — pod and container security contexts, ServiceMonitor (off by default, gated on the Prometheus Operator API), PDB maxUnavailable, explicit namespaces, and a render-time guard that fails clearly if capacityBuffer is enabled without the CRD.

Two fork defects fixed

  1. priorityClassName was hardcoded to gmp-critical. On a cluster without Google Managed Prometheus the API server refuses every pod, and because the failure sits at the ReplicaSet level helm install still reports success while the Deployment sits at 0/2. Now a value, defaulting to gmp-critical, so GKE behaviour is unchanged.
  2. Pod anti-affinity matched app.kubernetes.io/name: karpenter, but this chart's name is karpenter-gcp, so it matched nothing and both replicas could land on the same node. Now uses the selectorLabels helper.

Reviewing the diff

Most of the CRD line count is a reformat: the fork had re-indented upstream's controller-gen output, and this takes it verbatim so future syncs are a plain copy. docs/index.yaml churns broadly because helm repo index regenerates every entry — that is the existing release process, not a mistake.

Testing

Verified on a 2-node minikube cluster:

  • Controller reads the location correctly in both directions — us-central1"location is a region", us-central1-b"location is a zone, extracting region" — failing only on absent GCP credentials, which is correct off-GCP
  • Both replicas scheduled onto separate nodes, confirming the anti-affinity fix
  • Reinstall with no gmp-critical on the cluster brings both pods up, confirming the priority class fix
  • CRDs accept the full v0.6.0 API surface server-side
  • All three documented breaking changes reproduce: spec.tags rejected as an unknown field, malformed alias rejected by CEL, reserved metadata.kube-env rejected
  • Security contexts apply and the controller runs under readOnlyRootFilesystem: true
  • RBAC grants list/watch/patch on nodeoverlays; CapacityBuffer guard fails without the CRD and renders with it
  • helm lint clean

Not tested: real node provisioning. That needs a live GKE cluster with IAM. Config plumbing, CRD schemas, RBAC and scheduling are verified; the GCP path beyond credential acquisition is not.

Upgrade notes

This is breaking for existing users, documented in the chart README. Briefly: settings.locationsettings.clusterLocation; NodePool instance-family values must drop the shape suffix (n4-standardn4); GCENodeClass tagslabels; imageSelectorTerms[].alias deprecated; the broad compute.admin/container.admin IAM roles should be replaced with upstream's minimal custom role.

Note that Helm never upgrades CRDs in crds/, so existing installs must apply them manually before helm upgrade. The README documents this.

The chart tracked karpenter-provider-gcp v0.0.1 and was 11 releases
behind. The controller env var LOCATION was renamed upstream to
CLUSTER_LOCATION, so recent images started with no location configured.

Sync CRDs to v0.6.0, add the NodeOverlay CRD and its RBAC, expose the
six feature gates and the new controller settings, and add the pod and
container security contexts, ServiceMonitor, and PDB maxUnavailable.

Also fixes two defects in the local fork: priorityClassName was
hardcoded to gmp-critical, which blocks pod creation on any cluster
without Google Managed Prometheus, and the pod anti-affinity selector
matched a label the chart never sets, so replicas could stack on one
node.
Live GKE testing showed every instance create failing with
"Error 400: Invalid value for field 'resource.disks': ''. No disks are
specified." The CRD neither requires disks nor defaults it, so the
NodePool reports Ready while provisioning fails at the NodeClaim level.
…urable

The two defects found in this chart both came from the fork freezing a
value upstream exposes: priorityClassName blocked pod creation off-GKE,
and the anti-affinity selector named a label the renamed chart never
sets. The same shape of risk remained in image.repository, pullPolicy,
healthProbe.port, strategy, terminationGracePeriodSeconds, tolerations
and imagePullSecrets.

Restore all of them as values, keeping every current default so rendered
output is unchanged. healthProbe.port stays 8001 rather than adopting
upstream's 8081 so the port does not move under existing installs.

The anti-affinity default stays in the template using selectorLabels
instead of moving to values as a literal, so it cannot drift from the
chart name again. Missing values keys vs upstream drop from 15 to 4.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant