fix(certificatee): persist validated haproxy certificates

[ananthb]

Summary

• validate Vault certificates before certificatee writes anything to HAProxy
• persist certificatee updates through HAProxy storage with skip_reload=true, then update runtime for immediate effect
• delete already-expired Vault certs when certificator renewal fails so stale material cannot be replayed

Why

HAProxy reloads can lose runtime-only certificate updates. If certificatee only writes /runtime/ssl_certs, a restarted or reloaded LB can serve stale/default certs until the next certificatee sync. This makes certificatee write durable storage first and runtime second.

Validation

• /usr/local/go/bin/go test ./...

No Linear ticket - incident follow-up.

[github-actions]

Code Coverage Report

Total Coverage: total: (statements) 38.0%

Coverage by function

github.com/vinted/certificator/cmd/certificatee/health.go:27:		newCertificateeHealthChecker				100.0%
github.com/vinted/certificator/cmd/certificatee/health.go:38:		certificateeHealthMaxSyncAge				75.0%
github.com/vinted/certificator/cmd/certificatee/health.go:46:		Check							87.5%
github.com/vinted/certificator/cmd/certificatee/health.go:78:		MarkEndpointSyncSuccess					80.0%
github.com/vinted/certificator/cmd/certificatee/health.go:88:		lastSync						100.0%
github.com/vinted/certificator/cmd/certificatee/helpers.go:9:		createHAProxyClients					0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:24:		main							0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:76:		maybeUpdateCertificates					0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:92:		processHAProxyEndpoint					37.0%
github.com/vinted/certificator/cmd/certificatee/main.go:187:		setDataPlaneAPIVersion					100.0%
github.com/vinted/certificator/cmd/certificatee/main.go:197:		shouldUpdateCertificate					0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:220:		shouldUpdateForLiveExpiry				100.0%
github.com/vinted/certificator/cmd/certificatee/main.go:238:		shouldUpdateForSerialMismatch				100.0%
github.com/vinted/certificator/cmd/certificatee/main.go:248:		validateVaultCertificateForUpdate			0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:252:		validateVaultCertificateForUpdateAt			88.9%
github.com/vinted/certificator/cmd/certificatee/main.go:272:		parseVaultLeafCertificate				0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:281:		updateCertificate					0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:315:		buildPEMBundle						0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:341:		endsWith						0.0%
github.com/vinted/certificator/cmd/certificator/main.go:25:		main							0.0%
github.com/vinted/certificator/cmd/certificator/main.go:104:		deleteExpiredVaultCertificateAfterRenewalFailure	0.0%
github.com/vinted/certificator/pkg/acme/acme.go:27:			GetEmail						0.0%
github.com/vinted/certificator/pkg/acme/acme.go:32:			GetRegistration						0.0%
github.com/vinted/certificator/pkg/acme/acme.go:37:			GetPrivateKey						0.0%
github.com/vinted/certificator/pkg/acme/acme.go:42:			NewClient						0.0%
github.com/vinted/certificator/pkg/acme/acme.go:61:			setupClient						0.0%
github.com/vinted/certificator/pkg/acme/acme.go:80:			setupAccount						0.0%
github.com/vinted/certificator/pkg/acme/acme.go:118:			newAccount						0.0%
github.com/vinted/certificator/pkg/acme/acme.go:130:			getAccountKey						0.0%
github.com/vinted/certificator/pkg/acme/acme.go:159:			registerAccount						0.0%
github.com/vinted/certificator/pkg/acme/acme.go:177:			recoverAccount						0.0%
github.com/vinted/certificator/pkg/acme/acme.go:211:			saveAccount						0.0%
github.com/vinted/certificator/pkg/acme/acme.go:221:			saveKey							0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:18:	ObtainCertificate					0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:50:	GetCertificate						0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:63:	ParsePEMCertificate					0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:76:	DeleteCertificate					0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:81:	IsExpired						0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:87:	NeedsReissuing						0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:115:	arraysEqual						0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:129:	arrayContains						0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:138:	VaultCertLocation					0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:142:	storeCertificateInVault					0.0%
github.com/vinted/certificator/pkg/certmetrics/metrics.go:79:		StartMetricsServer					0.0%
github.com/vinted/certificator/pkg/certmetrics/metrics.go:104:		newHandler						100.0%
github.com/vinted/certificator/pkg/certmetrics/metrics.go:128:		PushMetrics						0.0%
github.com/vinted/certificator/pkg/config/config.go:73:			LoadConfig						0.0%
github.com/vinted/certificator/pkg/haproxy/client.go:57:		Error							0.0%
github.com/vinted/certificator/pkg/haproxy/client.go:63:		IsHTTPStatus						100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:68:		unexpectedStatusError					100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:77:		NewClient						100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:112:		NewClients						92.3%
github.com/vinted/certificator/pkg/haproxy/client.go:137:		Endpoint						100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:142:		doRequest						88.9%
github.com/vinted/certificator/pkg/haproxy/client.go:160:		parseAPITime						66.7%
github.com/vinted/certificator/pkg/haproxy/client.go:173:		getConfigVersion					66.7%
github.com/vinted/certificator/pkg/haproxy/client.go:238:		ListCertificates					100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:252:		ListCertificateRefs					90.9%
github.com/vinted/certificator/pkg/haproxy/client.go:294:		GetCertificateDetail					68.4%
github.com/vinted/certificator/pkg/haproxy/client.go:337:		UpdateCertificate					80.0%
github.com/vinted/certificator/pkg/haproxy/client.go:371:		UpdateStorageCertificate				90.9%
github.com/vinted/certificator/pkg/haproxy/client.go:390:		CreateCertificate					78.9%
github.com/vinted/certificator/pkg/haproxy/client.go:423:		EnsureStorageCertificate				50.0%
github.com/vinted/certificator/pkg/haproxy/client.go:441:		DeleteCertificate					92.9%
github.com/vinted/certificator/pkg/haproxy/client.go:465:		ExtractDomainFromPath					100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:485:		NormalizeDomainForVault					0.0%
github.com/vinted/certificator/pkg/haproxy/client.go:492:		StorageCertificateName					100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:498:		IsExpiring						100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:510:		NormalizeSerial						100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:543:		Error							100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:547:		Info							0.0%
github.com/vinted/certificator/pkg/haproxy/client.go:551:		Debug							100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:555:		Warn							0.0%
github.com/vinted/certificator/pkg/haproxy/client.go:560:		toLogrusFields						85.7%
github.com/vinted/certificator/pkg/vault/vault.go:18:			NewVaultClient						0.0%
github.com/vinted/certificator/pkg/vault/vault.go:27:			TokenLookupSelf						0.0%
github.com/vinted/certificator/pkg/vault/vault.go:39:			KVWrite							0.0%
github.com/vinted/certificator/pkg/vault/vault.go:53:			KVDelete						0.0%
github.com/vinted/certificator/pkg/vault/vault.go:66:			KVRead							0.0%
github.com/vinted/certificator/pkg/vault/vault.go:87:			vaultFullPath						0.0%
total:									(statements)						38.0%