[Adam-D-Lewis]

Summary

A PassthroughModel can now front an OpenAI-compatible server whose certificate comes from a private or corporate CA. Examples are llama.cpp's llama-server on a Mac Studio, or vLLM on another host on the LAN. A new optional field, spec.provider.tls.caConfigMapName, points the gateway at a ConfigMap of trusted CAs for that provider. Omitting it keeps today's behavior.

The PR also adds a Self-hosted servers guide with a llama.cpp walkthrough, a matching example manifest, and troubleshooting entries for upstream TLS failures.

Problem

The operator always renders the provider's BackendTLSPolicy like this:

validation:
  wellKnownCACertificates: System
  hostname: <spec.provider.hostname>

Envoy Gateway implements System as the CA file inside the Envoy proxy image (/etc/ssl/certs/ca-certificates.crt). That file holds only the public roots. The connection to the provider is always TLS, and no field selects a different trust source. So a self-hosted server on a private network can't be used:

• A public CA can't issue a certificate for a host that has no public DNS.
• A certificate from the organization's own CA fails verification.
• Mounting a different CA file into the shared gateway's proxy pods works around it, but that changes trust for every upstream on the gateway, restarts the proxies, and relies on an Envoy Gateway internal path. The provider docs already advise against patching the generated Envoy deployments.

Change

API. ProviderSpec gains an optional tls block:

spec:
  provider:
    hostname: macstudio.corp.example.com
    port: 8443
    credentialSecretName: macstudio-api-key
    tls:
      caConfigMapName: corp-ca

caConfigMapName is required inside tls and has a minimum length of 1. The provider resolver checks that it's a valid object name, so the webhook rejects values such as Corp_CA.

Rendering. When the field is set, the BackendTLSPolicy uses caCertificateRefs (group: "", kind: ConfigMap) in place of wellKnownCACertificates. Gateway API's CEL rules allow exactly one of the two, so the bundle replaces the public roots for that provider. Other PassthroughModels are unaffected. The operator already applies generated resources with a full Update, so switching an existing model between the two modes doesn't leave both fields behind.

Behavior that comes from Envoy Gateway, with nothing added in the operator. Each was checked in the v1.8.4 source:

• The CA is read from the ca.crt key, or from the ConfigMap's only key. So a trust-manager Bundle target (one key, usually ca-certificates.crt) can be referenced directly.
• BackendTLSPolicy is indexed by its referenced ConfigMaps, so editing the ConfigMap re-translates the policy. Valid CA updates need no restart.
• A missing ConfigMap or invalid key layout is reported on the policy status as Accepted=False / NoValidCACertificate and ResolvedRefs=False / InvalidCACertificateRef. Malformed PEM is different: in live testing with Envoy Gateway 1.8.1, the policy could report accepted while Envoy rejected the TLS secret. An invalid replacement of a working bundle could retain the previous valid trust. The troubleshooting guide explains how to check proxy logs and verify rotation with a real request.

Docs.

• New self-hosted.mdx page, added to the sidebar after AWS Bedrock. It leads with steps that don't depend on the framework:

  • how the two TLS hops work
  • the requirements: reachable, a DNS name rather than an IP (Envoy Gateway rejects IPs and single-label names), a certificate for that name, an API that requires a key
  • generic server setup: certificate on the host, HTTPS from the serving framework or a reverse proxy, API key, kept running
  • connecting it to the cluster: Secret, CA ConfigMap, the CRD-upgrade note, the PassthroughModel
  • verify

  A caution at the top lists the single-server trade-offs: no failover or load balancing, overload left to the serving framework (the pack adds no queueing or rate limiting), the fixed 120s timeout, and no health checks. An Example: llama.cpp on a Mac Studio section at the end holds the specifics: Homebrew, the llama-server flags, launchd, sleep and firewall.

• examples/passthrough-llamacpp.yaml, embedded through ExampleCode, like the other examples.

• configuration.mdx: new row for spec.provider.tls.caConfigMapName.

• troubleshooting.md: two new entries, "Upstream connection fails (HTTP 503)" and "A CA bundle update does not take effect".

• One-line cross-links in quickstart.mdx, providers.mdx and architecture.mdx. The architecture resource table no longer says "System CAs" only.

Upgrade impact

• Additive and optional. It adds a new optional field to v1alpha1, with no version bump, no conversion, and no change to defaults. An existing PassthroughModel without tls renders the identical BackendTLSPolicy, so upgrading changes nothing on the cluster for it.
• The CRD has to be updated before anyone uses the field. Argo CD applies the chart's crds/ directory on sync unless skipCrds is set, so Argo users get the new schema with the upgrade. Plain helm upgrade does not; that path is covered by Upgrading (added by docs: one Upgrading page for routine and release-specific steps #210), and the new guide links there.
• Nothing breaks if the CRD lags. A new operator against the old CRD behaves exactly as before, because tls is simply absent. The CRD update is a prerequisite only for using the field, and the new guide says so.
• Release notes. Please label this type: enhancement 💅🏼 so the auto-generated notes list it under Features. The notes carry only the PR title, so if the release body is edited by hand, a one-liner would help: "PassthroughModel can trust a private CA (spec.provider.tls.caConfigMapName); Helm users must update the CRD to use it, see Upgrading."
• Rollback. Downgrading to a release without the field takes the trust setting away, and an affected model goes back to the public roots, so TLS to a private-CA server fails. Remove the PassthroughModel before downgrading, or accept that it will be down.

Files changed

• operator/api/v1alpha1/passthroughmodel_types.go: ProviderTLS type and ProviderSpec.TLS
• operator/api/v1alpha1/zz_generated.deepcopy.go: regenerated
• operator/config/crd/bases/llm.nebari.dev_passthroughmodels.yaml: regenerated
• charts/nebari-llm-serving/crds/passthroughmodel-crd.yaml: byte-identical copy of the base CRD
• operator/internal/provider/provider.go: Resolved.CAConfigMapName and validation
• operator/internal/controller/reconcilers/passthrough.go: BackendTLSPolicy trust source
• operator/internal/provider/provider_test.go, operator/internal/controller/reconcilers/passthrough_test.go, operator/internal/controller/backendtlspolicy_trust_test.go (new), operator/internal/webhook/v1alpha1/passthroughmodel_webhook_test.go: tests
• examples/passthrough-llamacpp.yaml: new
• docs/src/content/docs/self-hosted.mdx: new
• docs/astro.config.mjs, docs/src/content/docs/{configuration,troubleshooting,quickstart,providers,architecture}.*, docs/test/docs-examples.test.mjs

Testing

All runs were local on this branch.

• controller-gen v0.19.0 (the Makefile pin) for object and crd. The only generated changes are the new type and field.
• Full operator suite with envtest (Kubernetes 1.34.1), the same command as make test: all packages pass. That includes the new webhook specs, which admit examples/passthrough-llamacpp.yaml from disk and reject Corp_CA through a real API server.
• Switching modes both ways: TestBackendTLSPolicyTrustSourceSwitchesBothWays drives one model System → CA → System → CA through BuildPassthroughResources and the shared apply helper (fake client), and checks the stored policy holds exactly one trust source after each step. I confirmed it bites: making the builder emit both sources fails it at the first switch to the CA. TestBuildPassthroughResourcesBedrockTrustsConfiguredCA covers Bedrock with a private endpoint and a CA ConfigMap.
• golangci-lint v2.4.0 (the CI pin) on operator/: 0 issues.
• key-manager go test ./...: passes. It imports the operator module.
• Docs: npm test passes (31 tests, including new self-hosted checks that keep the --alias and the declared id in step). Rerun after merging main (with docs: one Upgrading page for routine and release-specific steps #210) into this branch and after the troubleshooting update in de163f6: npm test still passes, and a build with the production base (BASE=/llm-serving-pack/) passes scripts/check-links.sh with LINKS_OK. The self-hosted guide's /upgrading/#routine-upgrade link points at an existing anchor.
• One-off admission check, not committed. I ran envtest with the Gateway API v1.5.1 standard CRDs and created the rendered BackendTLSPolicy in both modes. Both were accepted, and the stored object kept caCertificateRefs: [{group: "", kind: ConfigMap, name: ...}]. A policy with both trust sources was rejected with must not contain both CACertificateRefs and WellKnownCACertificates, which confirms the CEL rule the builder relies on. CI's TestGatewayAdmission loads only the AI Gateway CRDs, so it does not cover BackendTLSPolicy.

Live data-plane verification

Tested operator commit 81571fb on a disposable Kind 0.32.0 / Kubernetes 1.36.1 cluster using Helm 4.3.0, Envoy Gateway 1.8.1, Envoy 1.38.1 and Envoy AI Gateway 1.1.0. Later commits change only documentation, plus one code comment that came in with the merge of main.

Requests traversed the operator-generated route and TLS policy to a temporary, API-key-protected HTTPS nginx proxy in front of an existing Prism llama.cpp router running Ternary-Bonsai-2-27B. Both client-facing and upstream certificates were verified; the client did not use --insecure.

• Correct CA: normal and streaming chat returned HTTP 200 and model content TLS_OK; the stream completed with [DONE].
• System roots only, wrong CA, and hostname mismatch: HTTP 503. Proxy logs confirmed certificate verification and SAN mismatch failures.
• Missing CA ConfigMap: HTTP 500 and the documented policy reference errors.
• System → CA → System → CA switches: requests followed the trust setting; the generated policy contained exactly one trust source.
• CA rotation: a new server certificate failed against the old CA, then succeeded after updating the existing ConfigMap. Envoy pod UIDs and container restart counts were unchanged.
• Malformed PEM: Envoy rejected the TLS secret despite accepted policy status. Updating a working bundle with invalid PEM retained previous valid trust, confirmed after restarting the upstream proxy and making new connections. Restoring the valid bundle recovered normally.

Limits: this used TLS termination at nginx, not llama.cpp's native TLS listener. It did not exercise production DNS/ACME, Keycloak login, or key-manager key issuance; a test client key was inserted into the operator-managed Secret using the documented local-dev workflow. The test scripts/evidence were retained locally and are not committed CI coverage.

The local dev setup encountered a Gateway API standard/experimental CRD channel conflict. The test preserved the standard CRDs, applied Envoy-specific CRDs separately and installed the Envoy chart with --skip-crds; no admission policy was disabled. The setup defect and tested workaround are tracked in #213.

Notes for reviewers

• Field shape. I used a single name, caConfigMapName, to match credentialSecretName, rather than mirroring Gateway API's caCertificateRefs list. One trust-manager bundle can hold any number of CAs. If reviewers prefer the list shape, or want Secret support (which Envoy Gateway also accepts), the change is local to ProviderTLS and buildProviderBackendTLSPolicy.
• Out of scope: the fixed 120s request timeout on passthrough routes. A slow self-hosted model can hit it, so the guide lists it among the trade-offs. Making it configurable is tracked in #209.
• Status. The operator does not copy BackendTLSPolicy status into the PassthroughModel, so Ready means "resources applied". An informational CA ConfigMap resolution condition/watch is tracked separately in Add informational CA ConfigMap resolution status to PassthroughModel #212. Docs distinguish reference resolution from actual certificate loading and point readers to proxy logs and live requests as well as policy status.

[github-actions]

📄 Docs preview for feat/passthrough-provider-ca-trust:
https://feat-passthrough-provider-ca.llm-serving-pack.pages.dev

[tylerpotts]

Thanks @Adam-D-Lewis, this is a clean, well-scoped change. The new field takes the same route as hostname and port (ProviderSpec → provider.Resolved → builder), the webhook enforces it through the same Resolve call, and building exactly one trust source plus the full-Update apply means switching modes can't leave both fields behind. The PR description is a model of how to write one.

What I verified at 04749ec:

• Operator build/vet are clean, and the provider, reconcilers, controller and webhook suites pass with envtest (Kubernetes 1.34.1). Docs npm test passes (31 tests).
• Mutation checks: making the builder emit both trust sources fails TestBackendTLSPolicyTrustSourceSwitchesBothWays at step 1 and both TrustsConfiguredCA tests. Removing the resolver's name check fails the Corp_CA webhook spec. The new tests catch what they're meant to.
• The chart CRD is byte-identical to config/crd/bases.
• Upstream claims in the guide hold up against Envoy Gateway v1.8.4 and llama.cpp v0.7.0: ca.crt-or-only-key, re-translation on ConfigMap edits, the NoValidCACertificate/InvalidCACertificateRef reasons, IP/single-label rejection, the llama-server flags, and Homebrew building with OpenSSL.
• Not run: a live handshake through Envoy Gateway, same as you noted.

Before merge: two doc lines describe Envoy behaviour that doesn't happen (inline below). Also, scripts/check-links.sh currently fails on self-hosted/index.html -> /upgrading/. That's the expected dependency on #210, whose ## Routine upgrade anchor does exist. The "LINKS_OK" line in the PR description predates the commit that added the link, so CI will be red until #210 merges and this branch is rebased.

Follow-up, not blocking: the credential Secret has an informational UpstreamCredentialResolved condition, backed by an index and a watch. The CA ConfigMap is the second user-supplied reference and has no equivalent, so a missing ConfigMap only shows up on <name>-backend-tls. The operator ClusterRole already allows get/list/watch on configmaps, so an UpstreamCAResolved condition would need no new RBAC. Your status note explains the current choice, so a follow-up issue is fine.

Approving. Please make the two doc fixes before merging.

[tylerpotts]

The 503 is right, but CERTIFICATE_VERIFY_FAILED never reaches the client. Envoy v1.38.4 (the default proxy image in Envoy Gateway v1.8.4) hides the transport failure reason from the response body by default: envoy.reloadable_features.hide_transport_failure_reason_in_response_body is a RUNTIME_GUARD (default true, source/common/runtime/runtime_features.cc:69), checked in source/common/router/router.cc:1612-1621. Envoy Gateway doesn't override it. The client gets:

upstream connect error or disconnect/reset before headers. reset reason: connection failure

The TLS_error:...CERTIFICATE_VERIFY_FAILED detail only shows up in the proxy access log (%RESPONSE_CODE_DETAILS%). Someone grepping their curl output for the string in this heading won't find it and may rule out TLS. Suggest quoting the real body in the heading and adding a step to check the Envoy proxy logs for CERTIFICATE_VERIFY_FAILED.

[tylerpotts]

Nit: in llama.cpp v0.7.0, --no-webui is a deprecated alias of --no-ui (common/arg.cpp:3491), and the server logs a deprecation warning. It still works. Keeping it is reasonable if you'd rather not break older Homebrew builds, so take it or leave it.

[Adam-D-Lewis]

@tylerpotts Review follow-up: the requested doc corrections and the optional flag nit are addressed in 81571fb and 0300853:

• TLS troubleshooting now distinguishes the client response from the certificate-verification detail in proxy logs.
• SHA-256 is a recommendation, not a claimed gateway requirement.
• The llama.cpp example uses --no-ui.

The previously missing live data-plane test is now done. Operator commit 81571fb ran on Kind/Kubernetes 1.36.1 with Envoy Gateway 1.8.1, Envoy 1.38.1, and AI Gateway 1.1.0. A real Prism llama.cpp router running Ternary-Bonsai-2-27B sat behind a temporary authenticated HTTPS proxy. Both TLS hops were verified; the client did not use --insecure.

Normal and streaming requests through the operator-generated route returned TLS_OK. Wrong CA, system roots against the private-CA server, and hostname mismatch failed verification. Missing ConfigMap produced the documented policy errors. Switching System → CA → System → CA worked, and replacing the server certificate and updating the existing CA ConfigMap restored inference without an Envoy pod/container restart.

One observed caveat is now documented: malformed PEM can leave the policy accepted while Envoy rejects the TLS secret; an invalid update to a working bundle can retain the previous valid trust. Rotation verification therefore needs proxy logs and a request against the new server certificate, not policy status alone.

These were local integration tests, not committed CI coverage. TLS terminated at the proxy in front of llama.cpp; native llama.cpp TLS and the full Keycloak/key-manager flow were not tested. The PR description now records these results and limits.

Follow-ups: #212 tracks the informational CA-reference status/watch you suggested. #213 tracks a separate dev-setup Gateway API CRD channel conflict found during testing.

All 31 local docs tests pass. The latest CI has 12 passing checks; docs still fails on the /upgrading/ dependency. #210 needs to merge first, then this branch needs updating and the docs checks rerun.

[tylerpotts]

Thanks @Adam-D-Lewis. I re-checked at 0300853. The two doc fixes I asked for before merge are done, so my approval stands. The new rotation and malformed-PEM guidance is a good addition from the live test.

Previous review, checked at 0300853:

Item	Status	Evidence
Troubleshooting heading quoted CERTIFICATE_VERIFY_FAILED, which never reaches the client	Fixed	troubleshooting.md:301 now quotes the client body and sends readers to the proxy logs
"SHA-256 or stronger" listed as a requirement in troubleshooting	Fixed	Removed from the checklist at troubleshooting.md:323
Same SHA-256 wording in the self-hosted guide	Fixed	self-hosted.mdx:52 now says "(SHA-256 or stronger recommended)"
Nit: --no-webui is a deprecated alias	Fixed	--no-ui at self-hosted.mdx:148 and :153
Follow-up: CA ConfigMap resolution condition	Tracked	#212

Merge order: #210 is approved and should merge first. I trial-merged it into this branch:

• The only conflict is the docs/astro.config.mjs sidebar. Resolving it as the description says gives Self-hosted Servers, Installation, Upgrading.
• On the merged tree, docs npm test passes (31), scripts/check-links.sh reports LINKS_OK, and self-hosted links to /llm-serving-pack/upgrading/#routine-upgrade under the subpath build.
• The only failure in this PR's current docs CI is the expected self-hosted/index.html -> /upgrading/ broken link. It should go green once this branch is updated after #210 merges.

New troubleshooting text, checked against upstream source (Envoy Gateway v1.8.4, which defaults to Envoy v1.38.4):

• The response body hides the transport failure reason by default (router.cc:1612-1621 and the runtime guard at runtime_features.cc:69).
• EG's default JSON access log is enabled and writes %RESPONSE_CODE_DETAILS% and %UPSTREAM_TRANSPORT_FAILURE_REASON% (internal/xds/translator/accesslog.go:46,48). That means the grep CERTIFICATE_VERIFY_FAILED matches. This holds as long as no EnvoyProxy overrides the access-log format, and this repo has none.
• Failed to load trusted CA certificates from <inline> is the real Envoy message for a bad inline CA (default_validator.cc:86-89). EG delivers the ConfigMap as inline bytes over SDS, so the path really is <inline>.
• Secret is not supplied by SDS is real too (ssl_socket.cc:26). It is a transport failure reason, not a standalone log line. Envoy replaces spaces with underscores in response_code_details, so this string only matches through the upstream_transport_failure_reason field. Your grep -E still catches it.
• The proxy container is named envoy (internal/infrastructure/kubernetes/proxy/resource.go:30).

Nit, non-blocking: troubleshooting.md:304 has readers pick <envoy-proxy-pod> out of a bare get pods, which also lists the controller. The same file already uses -l gateway.envoyproxy.io/owning-gateway-name=<gateway> (lines 191-192), so kubectl logs -l ... -c envoy would skip the copy-paste step. Passthrough routes go on the internal or external gateway, not necessarily nebari-gateway, so the placeholder should stay generic.

Not run this round: the operator suites and mutation checks. 81571fb and 0300853 change only docs, so last round's results at 04749ec still apply. I didn't run a live data-plane test; your Kind run covers that.

[Adam-D-Lewis]

@tylerpotts Took the nit in de163f6: both proxy-log commands in the upstream TLS entries now select pods with -l gateway.envoyproxy.io/owning-gateway-name=<gateway>, and the text names the chart defaults for the external and internal gateways. I also added --tail=-1: with a label selector, kubectl logs otherwise returns only the last 10 lines of each pod, even with --since, so the grep could miss the failure. #210 is merged into the branch, and the docs tests, build and link check pass locally.