Pantera 2.3.0 — S3 storage rebuild (1000+ req/s R+W), per-format API completeness, HA correctness & security hardening - #58
Open
aydasraf wants to merge 141 commits into
Open
Pantera 2.3.0 — S3 storage rebuild (1000+ req/s R+W), per-format API completeness, HA correctness & security hardening#58aydasraf wants to merge 141 commits into
aydasraf wants to merge 141 commits into
Conversation
Implementation specs for the 2.3.0 release under docs/specs/2.3.0/: workstreams WS1-WS7, per-format WS4 sub-specs, the signed-off security wire-or-delete decisions, and the synthesized gap analysis + API-completeness matrix that ground them.
With cooldown active the packument ETag was derived from the immutable upstream metadata hash, so a client that cached the filtered packument kept revalidating to 304 and never saw a version that had aged out of (or been released from) cooldown until it cleared its cache. The ETag is now computed from the filtered bytes actually served (matching the Maven adapter) and the raw-hash early-304 is skipped while cooldown is active. Maven/PyPI/Docker were unaffected.
Hosted @latest now selects by semver (was lexicographic string order and compared the full key, not the version). Remove the non-functional .ziphash sidecar wiring and its misleading Maven-equivalent-integrity claim (GOPROXY defines no such sidecar; an h1: dirhash is not a zip-byte digest). Docs: replace the no-op GONOSUMCHECK/GONOSUMDB guidance with GOSUMDB=off / GOPRIVATE.
Local artifact HEAD now emits Content-Length + Last-Modified and the proxy cache-hit HEAD emits the Last-Modified its javadoc promised. Group maven-metadata.xml .sha256/.sha512 are recomputed over the exact bytes the group serves, like .sha1/.md5. Remove the orphaned RepoHead + its ITCase and the unreachable maven-plugin/ejb EXT branches.
Yank/unyank now require write permission on the target repository (previously any authenticated token could yank any repo); denial is 403, fail-closed, no mutation. Twine uploads verify the declared sha256_digest (reject with checksum_mismatch on mismatch) and reject a re-upload of an existing distribution filename with 409.
…EST_INVALID Proxy blob cache-store re-hashes via CheckedBlobSource so a corrupt/truncated upstream body fails the cache write instead of being cached and re-served under a digest it does not match; the client still receives and independently verifies the tee'd bytes. tags/list and _catalog emit Link: rel=next when truncated. PUT-by-digest mismatch fails with an explicit DIGEST_INVALID. Upload cancel returns 204.
npm cooldown ETag, Go @latest/ziphash/sumdb docs, Maven HEAD/group checksums, PyPI upload validation + yank authz, Docker blob verify/pagination/digest errors.
…agate authz/settings WS2.1: token revocation is DB-durable + Valkey-accelerated — hydrated on boot, reconciled on a short poll, pub/sub payload carries the token's real remaining TTL, so a node restarting after a revocation no longer re-honours it. WS2.2: the artifact-events drain runs on a per-node ScheduledExecutor, never through cluster-shared Quartz; QuartzJob no longer deletes a job on a node-local-dependency miss and the blanket scheduler.clear() becomes a scoped stale-group purge — no lost audit/index rows. WS2.3: the policy cache is a PublishingCleanable (expireAfterWrite backstop) and breaker/bulkhead settings broadcast over Valkey, so authz and resilience thresholds propagate cluster-wide. Extracts a CacheBroadcast interface; corrects the CLAUDE.md license:format command and stale HA docs.
…re search to the DB index dist-tags persist in a durable <pkg>/.dist-tags.json that the generated packument merges over the computed latest, so dist-tag ls/add/rm, publish --tag, deprecate, and single-version unpublish work for published packages (they previously read a meta.json publish never wrote and 404'd). npm search now queries the shared, already-populated ArtifactIndex; the never-populated in-memory index is deleted.
… upstream URLs The php-proxy root /packages.json was routed through the per-package cache slice, which mangled it and 404'd, so a standalone proxy could not bootstrap composer install; it now fetches the raw remote. Every top-level root URL (metadata-url, search, list, security-advisories, ...) is rewritten back to Pantera (unknown absolute URLs dropped fail-closed), closing the cache/cooldown/auth bypass in proxy and group mode.
Flipping a release's yank sidecar left the served /simple/ index frozen at upload time, so pip/uv never saw data-yanked without a re-upload. yank/unyank now regenerate the package HTML+JSON index and invalidate the negative/filtered caches, with a no-op guard so yanking an unpublished version never fabricates a phantom index.
The Go proxy fetched @v/list and @latest live on every request and never cached them, so go get / go list -m -versions broke on any upstream blip even for fully-cached modules. Both now read through a shared, single-flighted 12h TTL cache that serves the last-known-good document when upstream is unreachable; a hosted publish invalidates the base cache.
…lter) Pushing a manifest with a subject indexes it under a listable referrers prefix; GET /v2/<name>/referrers/<digest> serves a real OCI image index with artifactType filtering + OCI-Filters-Applied, and the push response carries OCI-Subject. cosign (OCI mode), notation, oras attach/discover, and SBOM attachment now work; the endpoint was previously a permanent empty stub.
…1.0) Extracts a backend-agnostic BlobStore + Presigner interface (in the SDK-free storage core) that later disk-primary caching and presigned direct-download build on. S3Storage becomes the reference impl (existing Storage behaviour unchanged; BlobStore methods delegate) with SigV4 presigning via the AWS SDK's own presigner. The S3 backend now supports S3-API-compatible object stores (MinIO, Cloudflare R2, Backblaze B2, Wasabi, Ceph/RADOS, GCS S3-interop) via endpoint/region/path-style config, with a configurable object storage-class; S3ExpressStorageFactory folds into a thin S3StorageFactory subclass.
…a cache hit load()'s switchIfEmpty(RxFuture.single(remote.get())...) evaluated remote.get() as a plain method argument — eagerly, while building the reactive chain, on every call including cache hits — so a side-effecting Remote fired even when the artifact was served from cache. Wrapped in Single.defer(...) so the upstream fetch runs only when the fallback is actually subscribed (a genuine miss). Behaviour-preserving: pantera-storage-core + pantera-core (incl. all BaseCachedProxySlice tests) suites green.
… (WS1.1) Adds an in-memory StorageIndex (rebuilt from a boot-time disk scan of the cache's .meta sidecars) and a CachedBlobStorage that composes a local disk tier, the index, and the WS1.0 BlobStore cold tier. exists/metadata/list are answered from the index and value() streams from disk via the NIO fast path — a cache hit makes ZERO blob-store round-trips (vs DiskCacheStorage's 1-2 HEADs per hit); a miss single-flights exactly one BlobStore.get. Selected opt-in via cache.mode: index; the default (DiskCacheStorage) is unchanged. Cold-fill-waits-for-full-object and no-eviction are documented limitations addressed by later WS1 phases (streaming, eviction). Self-contained single-flight (not pantera-core's SingleFlight) to keep the storage layer free of the pantera-core dependency edge.
NpmStarRepository and MetadataEnhancer.enhanceWithStars were never wired into any route (WS4a hygiene decision, 2.3.0 security-integrity review): delete the dead code rather than ship an inert `npm star`/`unstar` claim. enhance()'s always-empty `users` object is kept since clients expect the field to be present.
…t cold fetches (WS4a S7)
Dist archives fetched through a php-proxy are now verified against the
packument's declared dist.shasum (SHA-1 — Composer's ArchiveDownloader
verifies with hash_file('sha1', ...), despite the field's confusing name)
before anything lands in the cache. A mismatch is rejected (502 +
X-Pantera-Fault, cache stays empty) instead of silently caching corrupted
bytes. Concurrent cold requests for the same archive now single-flight to
one upstream call instead of one per concurrent request. Hosted publish
computes and stores dist.shasum on the archive it actually serves, so
downstream Composer clients (and a Pantera instance mirroring this repo
as a proxy) can verify it too.
The dead WI-07 integrity path this replaces verified a phantom .sha256
sidecar Composer never serves, on a route (/p2/<vendor>/<pkg>.json) that
a rewritten dist URL (/dist/<vendor>/<pkg>/<version>.zip) never matches,
so it never ran in production; its test coverage is removed with it. The
real, reachable path now lives in ProxyDownloadSlice, which already
resolves the (frequently cross-host) dist URL correctly.
…& dist-tags proxying, single-version, ping, honest audit
Implements the remaining WS4-npm 2.3.0 items (npm search and dist-tags/
deprecate/unpublish landed in a prior commit):
- WS4a security decision S1 (WIRE): publish now signs each version's
dist block with the registry's own ECDSA P-256 keypair (generated once,
persisted per-repo), the same way the public npm registry signs
packages. GET /-/npm/v1/keys serves the public key so `npm audit
signatures` verifies for real. `npm publish --provenance` attachments
are distinguished from tarballs and routed to a durable attestation
sidecar instead of being silently dropped or mis-stored; GET
/-/npm/v1/attestations/<spec> serves the bundle back.
- GET /<pkg>/<version> and GET /<pkg>/latest now return a version's
manifest directly (resolved through the durable dist-tags sidecar),
instead of 404ing through the packument catch-all.
- GET /-/ping and a real GET /npm registry-info response replace the
empty-body stub; HEAD is now supported on packument/tarball routes.
- LocalAuditSlice consumes its request body (previously leaked) and
answers with the correct shape for the audits vs advisories/bulk
endpoints instead of a bare {} for both.
- npm-proxy repositories forward /-/v1/search and dist-tags GETs
upstream instead of 404ing them through the packument route.
- /-/npm/v1/tokens (list/create/revoke) and /-/npm/v1/user (profile)
are wired for non-JWT-only repositories via the existing
StorageTokenRepository/StorageUserRepository; JWT-only repositories
answer honestly with 501 rather than a silently-empty token list.
POST /-/v1/login is wired to the existing OAuthLoginSlice for
JWT-only web login.
Documents the newly-working dist-tags, search, audit, signing, and
token/profile surface in the npm user guide.
…upport - GET /p2/available-packages.json and GET /packages/list.json (backing composer show -a / composer search) no longer 404: served from the shared ArtifactIndex for local/hosted php repositories, and as a live passthrough to the upstream for php-proxy (both were advertised in the root document but unrouted, previously falling into ProxyDownloadSlice's catch-all forward-to-remote). - CachedProxySlice issues a conditional If-Modified-Since request when revalidating already-cached package metadata (using the Last-Modified captured on the prior fetch); a clean 304 skips the merge/rewrite/save cycle instead of re-transferring and re-parsing the full body. - HEAD is now supported everywhere GET is (local package metadata, dist downloads, the new catalog surfaces, and the proxy dispatch as a whole), returning GET's status/headers with no body. RepositorySlices.java: additive — threads the already-shared ArtifactIndex into PhpComposer's read side via a new constructor overload; the existing 7-arg overload (and every other call site) is unaffected.
…ersion-JSON cooldown leak The cooldown-filter pipeline's Link model carried no yanked field, so every proxied pip (HTML) request silently dropped data-yanked regardless of whether cooldown was even active for that request. The JSON simple index emitted data-dist-info-metadata (the HTML attribute name, not a valid PEP 691/714 JSON key) so PEP 714 clients ignored it entirely. /pypi/<name>/<version>/json was deliberately unmatched by the cooldown detector and proxied straight upstream, leaking a cooldown-blocked version's metadata. Accept-header negotiation matched by substring, ignoring q-values and the latest+json/latest+html aliases. - PypiSimpleIndex.Link gains a yanked field; the parser reads data-yanked (HTML) / yanked (JSON) and the rewriter emits data-yanked so proxied yanks reach pip. - SimpleJsonRenderer emits the correct core-metadata JSON key, with dist-info-metadata retained as a legacy-client alias. - PypiJsonMetadataRequestDetector/PypiJsonHandler gain version-scoped detection and cooldown-filtered handling for <ver>/json (blocked -> 404); ProxySlice dispatches to it instead of falling through to the raw upstream passthrough. - SimpleApiFormat.fromHeaders rewritten to parse Accept media-ranges with q-values and the v1/latest json/html aliases instead of a bare substring match.
…, proxy/group HEAD - Hosted uploads now extract and persist the distribution's core metadata as a PEP 658 <file>.metadata sidecar, recording its sha256 in the sidecar; PySlice serves GET/HEAD for it, and the served index advertises both data-core-metadata (PEP 714) and the legacy data-dist-info-metadata attribute for the same value. - The PEP 691 JSON simple index now carries a top-level PEP-440-sorted versions[] array and a per-file size matching the stored artifact size. - IndexGenerator/SliceIndex/LegacyJsonSlice exclude .metadata sidecars from distribution-file enumeration: storage.list() is a raw string-prefix match, so a real file's own key also lists its .metadata sibling as a bogus extra "release file" without this filter. - Local repositories serve the legacy /pypi/<pkg>/json API used by poetry/pip-tools, synthesized from the persisted files + sidecars (LegacyJsonSlice). - PyProxySlice and CachedPyProxySlice answer HEAD for artifacts and index pages (missing -> 404, never 405). HEAD is translated to GET before any of CachedPyProxySlice's own caching branches run, so an uncached HEAD probe populates the cache with the real bytes instead of committing a phantom zero-byte artifact. docs/user-guide/repositories/pypi.md updated for all of the above.
WS4-go.4 / security decision S6 (WIRE). /sumdb/<name>/{supported,lookup,tile}
was previously unrouted in go-proxy and fell through the generic
fetchThroughCache path, caching the raw request as a bogus artifact.
GoSumdbHandler now proxies these to the same upstream(s) a go-proxy
repository already fetches modules from, per the GOPROXY-protocol
convention: lookup/tile responses are content-addressed and cached
immutably (zero upstream calls once warm, offline-safe), supported is
probed live. Clients can leave GOSUMDB at its default instead of
disabling checksum verification. Local ("go") and go-group repositories
have no upstream to proxy to and continue to honestly 404 the path.
Docs (go.md, jfrog-migration.md, troubleshooting.md) updated to stop
recommending GOSUMDB=off / the removed GONOSUMCHECK/GONOSUMDB no-ops as
the default guidance, and to document /sumdb/* behavior per repo mode.
Adds the pantera.go.sumdb.cache hit/miss counter (bounded repo_name/
kind/result tags, guarded by MicrometerMetrics.isInitialized()) with a
Grafana panel in pantera-cache-storage.json.
WS4-go.5: GoListHandler.blockedVersions evaluated cooldown for every parsed version unbounded, unlike GoLatestHandler's existing 50-version cap. Now sorts candidates by semver descending and evaluates only the newest MAX_VERSIONS_TO_EVALUATE (mirroring GoLatestHandler); versions beyond the cap are still served, only the cooldown-service fan-out is bounded. WS4-go.6: Go escapes uppercase module-path segments as `!` + lowercase on the wire (github.com/BurntSushi/toml -> github.com/!burnt!sushi/toml). That escaped form was written verbatim into the ArtifactEvent consumed by the DB/index/search/audit trail, in both the proxy path (GoProxyPackageProcessor) and the hosted upload path (GoUploadSlice). Both now decode the module name before constructing the ArtifactEvent; the storage/event key stays escaped since that is what `go get` actually requests and expects served back.
WS4-go.7. Goproxy.java (270 lines) was referenced only by its own GoproxyTest/GoproxyITCase — hosted publish uses GoUploadSlice, proxy caching uses CachedProxySlice. Removes the class and its tests, and corrects the goproxy/http package-info wording that described "Goproxy files" so it no longer reads as a reference to this class.
In cache.mode: index, save() now defaults to write-back: bytes land on the local disk tier and are acked from local durability, the index records PENDING_WRITE (persisted to the .meta sidecar), and a bounded pool of dedicated uploader threads drains the queue to the blob store with retry/backoff. A crash before drain replays pending writes on the next boot (the disk file is the payload, the PENDING_WRITE sidecar is the queue record). A bounded admission gate rejects new writes with WriteBackSaturatedException before any disk write, so a saturated queue cannot grow the cache unbounded; ProxyCacheWriter degrades gracefully on a proxy fill (client already teed), hosted uploads map it to 503 + Retry-After. Per-repo cache.write-through: true restores the prior synchronous semantics. Boot-replay uploads bypass the admission gate (already-durable bytes, not new admissions) and must NOT release a permit they never acquired; the success-path release now runs synchronously on the uploader thread, made deterministically testable via a boot-replay barrier. Config + docs (storage-backends, configuration-reference). storage-core (430), s3 (102), and pantera-core (ProxyCacheWriter) suites green; PMD + license clean.
Upload.append() no longer 405s on a second chunk. Each PATCH chunk stages under an offset-keyed slot (ordered independently of storage listing order) instead of the old single-chunk-only "second chunk throws" guard, and PUT finalizes by assembling every chunk in order and verifying the assembled bytes against the claimed digest via CheckedBlobSource (the same digest-verifying primitive the proxy blob cache uses). A chunk whose Content-Range start does not match the bytes already received is rejected with 416 Requested Range Not Satisfiable instead of silently accepted out of order. Docker/BuildKit's monolithic single-PATCH push is unaffected; oras/skopeo chunked pushers of very large layers now work.
Adds the missing OCI/Distribution delete surface: DELETE /v2/<name>/manifests/<reference> (202, removes the tag/digest link and any owned referrers-index entry) and DELETE /v2/<name>/blobs/<digest> (202, blob GC — deliberately independent of manifest delete, since blobs are content-addressed and may be shared). Both 404 on an unknown reference/digest and 405 on docker-proxy/docker-group, whose Manifests/ Layers composites reject delete with UnsupportedOperationException — deletes target the authoritative (docker) store only. New DockerActions DELETE permission bit, distinct from PULL/PUSH so existing roles don't silently gain it. Every delete is audited (artifact_delete) with captured trace/client-ip context.
…embers docker-group previously fell through the generic GroupResolver (first- 2xx-wins over each member's own HTTP Slice, no merge), so tags/list and _catalog silently returned only the first member's view. Wire a docker-group-specific branch instead: build each flattened member's Docker directly (mirroring how the "docker"/"docker-proxy" cases already construct AstoDocker/DockerProxy) and compose them with the existing MultiReadDocker, whose JoinedTagsSource/JoinedCatalogSource already give a deduplicated union - just never wired to docker-group. Manifest/blob GET keep first-2xx-wins via MultiReadRepo's prioritized walk (correct for content-addressed pulls); a member outage degrades to a partial union rather than a hard failure. DockerProxy.buildDocker() exposes the read-through Docker a docker-proxy repo serves without its HTTP-layer wrapping, for reuse as a group member. TrimmedDocker.registryName() now reports its own configured prefix instead of delegating to the origin - behavior-neutral for the existing docker/docker-proxy call sites (where the two already coincide), and necessary for docker-group so audit/log fields report the group's own name rather than an arbitrary member's.
…PARATE_THREAD) The class's blocking CompletableFuture.join() calls ignore interruption, so the default same-thread @timeout could not abort a load wedged under severe CPU starvation (observed when several concurrent local -T8 builds shared cores). Add a class-level SEPARATE_THREAD @timeout(90) and make the existing method @timeout SEPARATE_THREAD too, so a wedge fails fast and observably instead of stalling the reactor.
…file-safe The S3 load ITCase asserts wall-clock throughput and needs a MinIO container, but pantera-storage's unconditional failsafe binding ran it in the regular CI build, where a shared 2-core runner makes the floor fragile -- and its teardown used Files.walk, which throws UncheckedIOException (missed by the IOException catch) when a write-back daemon's .tmp staging file vanishes mid-walk, so a cleanup race failed the run. Gate it with @EnabledIfSystemProperty(run.load.test) so only the docs/slo/load-test wrapper (which now sets the flag) runs it; the CI build skips it. Rewrite deleteTreeQuietly with walkFileTree + visitFileFailed so teardown tolerates concurrently-vanishing files.
Two defects reported against 2.2.4 block corepack entirely:
- GET /<pkg>/<version> has no route outside hosted repos. PackagePath's
pattern swallows /pnpm/11.5.1 as a package name, and the upstream
version manifest is then run through AbbreviatedMetadata, which emits
{name, modified} with 200 OK — corepack destructures dist.tarball off
undefined.
- Tarball URLs are rooted at the winning group *member*, not the group
the client addressed; corepack rejects any tarball outside its
configured registry base. The no-config fallback additionally
hardcodes http:// and ignores X-Forwarded-*.
Spec covers both mechanisms (an X-Pantera-Client-Base header stamped in
TrimPathSlice, and a version-or-tag resolver generalising the existing
/latest shortcut), a real-client conformance sweep across all three repo
modes, and the 2.2.5 backport commit list.
Reading the request pipeline before planning against it invalidated the original design: - TrimPathSlice cannot carry the client base. Its recursion guard means only the first slice in a chain trims, and for a group member the stamp holds the *member's* path — which is exactly why GroupResolver strips X-FullPath. - SliceByPath is the single point that knows the client-facing path, the resolved repo key, and that repo's config, and it sits above GroupResolver — so group-wins falls out of stamp-if-absent with no member-level suppression. - Reading X-Original-Path (already stamped by ApiRoutingSlice) preserves the /api/<type> segment the client configured as its registry. Also records a third defect found while reading: the /latest shortcut documents that it applies no URL rewriting, so a proxy hands clients registry.npmjs.org tarball URLs — bypassing Pantera's cache and audit trail, and failing outright when air-gapped. The new resolver fixes it.
…e edge X-Pantera-Client-Base and X-Original-Path are internal-only signals: the former is stamped once by SliceByPath and echoed verbatim by the npm adapter into cached dist.tarball URLs; the latter feeds that derivation. Neither was ever stripped from inbound client requests, so a client sending its own value had it trusted outright - poisoning cached packument responses with an attacker-controlled host/path. SliceByPath.stamped() now unconditionally discards any inbound X-Pantera-Client-Base (case-insensitively - Headers.add's overwrite path compares names case-sensitively and cannot be used for this) before re-deriving the base for the addressed repository; group-wins is unaffected since SliceByPath is instantiated once and group members never re-enter it. ApiRoutingSlice now strips any inbound X-Original-Path on every request, matched or pass-through, before setting its own.
Delegates /<pkg>/<version> and /<pkg>/<tag> (including /latest) to VersionManifestResolver, which resolves against the cooldown-filtered packument with tarball URLs rewritten to the Pantera client-facing base. Removes the three single-purpose /latest methods (serveLatestManifest, resolveLatestFromRaw, buildLatestManifestResponse) that were passing upstream URLs through unchanged.
…dedicated ports - Local npm packument route (npm/http/DownloadPackageSlice) now honours the client-base header SliceByPath stamps for the repository a group actually addressed, matching SingleVersionSlice's precedence -- the last path where a local group member still leaked its own configured url:. - Emit Vary on every npm metadata response whose body embeds a derived base URL (local/proxy packument and single-version, 200s and their 304 twins), so a shared cache can no longer cross-serve one client's dist.tarball to another now that url: is optional. - Repositories bound to a dedicated port bypassed SliceByPath/ApiRoutingSlice entirely, so a client-supplied X-Pantera-Client-Base reached the slice unscrubbed there. Extracted the duplicated header-scrub helper to InternalHeaderScrub and apply it unconditionally on that path too via a new InternalHeaderScrubSlice decorator. - HEAD /<pkg>/dist-tags on npm-proxy fell through to the packument route (404) because only that route picked up HEAD; dist-tags and search passthrough routes now accept it too. - Corrected stale javadoc references to methods deleted earlier on this branch, and a stale test-method-name citation.
…ith admin UI Converts the env-only PANTERA_TRUST_FORWARDED_HEADERS flag into a DB-backed, hot-reloadable admin setting (trust_forwarded_headers), with the old env var kept as its fallback tier so existing deployments are unaffected. Adds a new client_base_host_allowlist setting closing the residual Host-header spoofing gap: even with forwarded headers untrusted, a client-supplied Host was still used verbatim to build cached absolute URLs (e.g. npm dist.tarball); a configured allowlist now rejects a non-matching Host exactly like an absent one. Empty/unset stays permissive by design and logs a startup WARN. Both settings: GET/PUT /api/v1/admin/client-base-url-settings, admin UI Settings page card, V137 migration, cross-node pub/sub broadcast.
…-less env support V137 unconditionally seeded trust_forwarded_headers/client_base_host_allowlist, so ON CONFLICT DO NOTHING never fired and an operator's PANTERA_TRUST_FORWARDED_HEADERS/PANTERA_CLIENT_BASE_HOST_ALLOWLIST env vars were silently shadowed on every upgrade. VertxMain also only installed ClientBaseUrlSettingsLoader inside sharedDs.ifPresent, so a DB-less boot never consulted those env vars at all, unlike the pre-2.3.0 static read. - V137 no longer seeds either key; documents them in comments only. - VertxMain installs the loader unconditionally (dao=null when DB-less), matching the loader's already-correct null-DAO -> env -> default chain. - Added an injectable env-lookup seam to ClientBaseUrlSettingsLoader so both regressions have deterministic, DB/Docker-free unit coverage. - Corrected stale javadoc/CHANGELOG wording that no longer matched behavior.
…ross all modes
Five npm client-compatibility bugs found by live-stack verification (WS8
final-verification-report.md):
- B1: a bare 2-segment scoped path (/@scope/pkg) matched the single-version
route ambiguously, 404ing instead of falling through to the packument
route on local repos. Tightened the route regex to mirror
SingleVersionSlice#parse's own disambiguation.
- B2: HEAD on an npm-proxy/group tarball fell through to the generic
FALLBACK 404, which CachedNpmProxySlice negative-cached regardless of
method, poisoning subsequent GETs of the same artifact. Routed HEAD to
the real asset handler and gated the negative-cache write on method so a
probe can never poison a real request. Pre-existing since v2.0.0, not a
regression from this branch (see fixwave-d-report.md for the diff
evidence).
- B3: npm ping was not wired for proxy/group repositories at all. Both now
answer directly from Pantera, no upstream/member round-trip.
- B4: the registry-root route was a hardcoded literal "/npm", unreachable
except by the coincidence of a repo literally named npm. Fixed to match
the actual repository root in all three modes.
- B5: proxy/group 404s for an unresolved version returned an empty body;
local mode's honest {"error":...,"package":...} shape is now reused
consistently regardless of cache state.
Full root-cause analysis, evidence, and exact test output per bug in
.superpowers/sdd/WS8-implementation-plan/fixwave-d-report.md.
Live verification of the 5 npm client-compatibility fixes in 6618f8b against a running 2.3.0 stack. B1-B4 (scoped packuments, HEAD tarball negative-cache poisoning, ping, registry root) confirmed fixed. B5 (honest 404 body on proxy/group for an unresolved version) is NOT fixed live despite the commit message's claim - reported as an open finding, not silently accepted. Full client x mode matrix (npm, pnpm, yarn classic, yarn berry, corepack) re-verified with cleared caches; all 11 acceptance criteria re-checked; DB-backed client-base settings confirmed hot-reloading without restart.
CachedNpmProxySlice already built the honest "version not found" JSON body (commit 6618f8b), but RaceSlice and GroupResolver both drained and discarded every 404 response's body on their all-targets-failed terminal, then completed with a bare bodyless 404 -- exactly the shape of loss RaceSlice already avoided for 403s. Every npm-proxy repository is raced through RaceSlice even with a single remote, and npm-group's sequential member walk hits the same pattern, so both modes served an empty body live despite passing unit tests that only drove CachedNpmProxySlice in isolation. RaceSlice now captures the first 404's body/headers via CAS (mirroring the existing firstForbidden handling) and forwards it instead of a bare notFound(). GroupResolver's WalkState captures a member's 404 body the same way and reuses it at both walk-terminal branches (including the non-authoritative/SKIP_HEADER path). New tests drive the actual production composition rather than a hand-built inner slice: CachedNpmProxySliceRaceCompositionTest wraps a real CachedNpmProxySlice in a real RaceSlice (npm-adapter has no network/DB dependency to fake), and two new GroupResolverTest cases drive the real sequential-walk terminal. Both fail on pre-fix code and pass post-fix; confirmed live against the local docker-compose stack for npm_proxy and npm_group, cold and warm. Also fixes pantera-main/docker-compose/.env.example, which could not boot the stack as shipped: POSTGRES_USER/KC_DB_USERNAME were uppercase PANTERA against the lowercase pantera role hardcoded in db/init/01-create-dbs.sql (Postgres folds unquoted SQL identifiers to lowercase but not connection-parameter usernames), and JVM_ARGS/ PANTERA_CONFIG referenced /var/PANTERA paths the image never creates (only lowercase /var/pantera) -- the GC log path in particular fails JVM startup outright when its parent directory doesn't exist.
… HEAD probe Bug 1: V136 unconditionally seeded upstream_breaker_* into auth_settings, so UpstreamBreakerSettingsLoader's DB -> env (PANTERA_UPSTREAM_BREAKER_*) -> default precedence never reached the env tier once a deployment ran that migration. V136 has already shipped, so it cannot be edited in place (Flyway checksums applied migrations); V138 instead deletes only the rows still holding their untouched V136 default, leaving any admin-customised value alone. VertxMain also only installed the loader inside sharedDs.ifPresent, so a DB-less boot never read the env vars at all -- it now installs unconditionally (dao=null when DB-less), matching the fix already applied to ClientBaseUrlSettingsLoader for the same bug class. Bug 2: GroupResolver -- the shared response() for every group type (maven/npm/pypi/docker/composer/go/gem/file) -- wrote a negative-cache entry off any member 404 with no method check. A HEAD probe (proxy, scanner, health check, routine existence check) that 404s could poison the cache and 404 every subsequent real GET for that artifact until expiry. Both cacheNotFound call sites now skip the write for HEAD, mirroring the existing guard in CachedNpmProxySlice.
…ing (WS8)
Fixes the reported 2.2.4 production bug: GET /<pkg>/<version> on npm proxy
and group repositories returned 200 with a {name, modified} stub, crashing
corepack's dist.tarball destructuring; and tarball URLs were rooted at a
group's winning member rather than the repository the client addressed.
Also lands two security fixes (internal client-base headers are no longer
client-spoofable; X-Forwarded-* and Host are gated behind DB-backed admin
settings), five pre-existing npm defects found by a live client sweep
(scoped packuments, HEAD tarball negative-cache poisoning, ping and
registry root on proxy/group, honest 404 bodies), and two shared-path
fixes (GroupResolver never negative-caches a HEAD probe; V136 no longer
shadows PANTERA_UPSTREAM_BREAKER_* env config).
Verified live against corepack, npm, pnpm, yarn classic and yarn berry
across local, proxy and group repositories.
V122 unconditionally seeded circuit_breaker_* rows into auth_settings, so CircuitBreakerSettingsLoader's DB -> env -> default precedence never reached the env tier once a deployment upgraded past V122. V139 removes only the rows still holding V122's untouched default, mirroring V138's fix for the identical upstream_breaker_* bug. CircuitBreakerSettingsLoader gains the same injectable env-lookup test seam as UpstreamBreakerSettingsLoader, and VertxMain now installs it unconditionally (dao=null on DB-less boots) instead of only inside sharedDs.ifPresent(...), which previously dropped PANTERA_CIRCUIT_BREAKER_* entirely on DB-less boots.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pantera 2.3.0 — storage-for-scale, per-format API completeness, HA correctness, and security hardening
The "bulletproof" major release. It rebuilds the S3 storage path to sustain ≥1000 req/s reads and writes against S3-compatible object stores, closes the gaps between advertised and actual API behavior across the major proxy/hosted formats, hardens the clustering and security posture, and fixes a set of genuine concurrency/correctness defects the layered gates surfaced along the way.
Everything below is on this branch, green under the full
mvn clean install -T8reactor (unit + PMD + license across all 33 modules) — 5,874 tests, 0 failures, 0 errors.Headline: storage for scale (WS1)
An index + async-write-back caching layer for the S3 backend, opt-in per storage via
cache.mode: index:BlobStore/Presignerabstraction — S3 + all S3-compatible stores (MinIO, R2, B2, Wasabi, Ceph, GCS-via-S3-interop) by config alone; thestorage-classis configurable.StorageIndexanswersexists/metadata/listwith zero blob-store round-trips, hydrated on boot from a disk scan; a disk-served read never contacts the object store.save()is acked from local disk, the S3 upload drains on a bounded pool with retry/backoff and a persistent (crash-replaying) queue; a saturated queue returns503rather than growing unbounded;cache.write-through: trueopts back into synchronous durability.Files.walk), cross-node pub/sub invalidation, an index-backed hosted-read slice (kills the oldexists()+value()double round-trip), blob-store + cache metrics, and presigned direct-download (302for Docker layer blobs via per-repodownload-mode).Release-gate load test (demonstrated, not asserted):
S3CacheLoadITCasedrives the realcache.mode: indexstack against a MinIO container. On a dev workstation:~876k reads in 15 s at a p99 of 5 ms is impossible against per-request MinIO round-trips — it demonstrates reads are served from the local disk tier via the index with zero blob-store contact. Numbers are from a single dev workstation, not a tuned benchmark rig; the point is the order-of-magnitude headroom over the gate, not the absolute figures. Harness + results:
docs/slo/load-test/.Per-format API completeness (WS4)
Made advertised-but-broken endpoints truthful across all six proxy/hosted formats: npm (package signing + provenance/attestations, tokens/profile, search & dist-tags through proxies, honest audit), Maven/Gradle (concurrency-safe
maven-metadata.xmlregeneration, PGP signature + checksum verification with quarantine, release immutability, Range requests), PyPI (PEP 658/700/714, HEAD), Composer (dist integrity, catalog/search, conditional GET), Go (checksum-databasesumdbproxy), Docker (manifest/blob DELETE, group tag/catalog aggregation, multi-chunk PATCH uploads,Acceptcontent-negotiation).HA correctness (WS2) & efficiency (WS5/WS6)
Token revocation survives restart in a cluster; authorization/threshold changes propagate cross-node; the artifact-event pipeline and the five proxy package-processors no longer lose data when a firing lands on a peer node. Cooldown-filtered metadata stays coherent under refresh; PyPI/Composer resolution surfaces are TTL-cached with serve-stale.
Security & correctness (from the adversarial review +
-T8gates)_users/_tokensexposure).verifyPgpis now fail-closed — an uploaded primary is quarantined (unresolvable, excluded from metadata) until a matching signature verifies, in either upload order.FromStorageCacheTest/SingleFlight/SchedulerDbTest/AsyncApiTestBasecontention issues that flaked the-T8reactor.Observability (WS7)
Grafana panels + a write-back-queue alert + runbook for the new storage/cache/write-back/eviction/download/sumdb metrics; every charted metric verified against its registration.
Completed after the initial scope (in this PR)
download-mode; metadata is never redirected (whitelist predicate per format). Conan stays stream-only (its per-item token-authenticated routes can't isolate a redirectable binary key safely).Accept-variant, Maven importerMetadataRegeneratorunification, WS5.5 PyPI filtered-index caching, and streamingcommitVerified(ref-counted temp lifecycle) all landed.ForkJoinPool.commonPool(), so the storage path isn't commonPool-starved on small/2-core hosts.Deferred to a follow-on (documented, not dropped)
BlobStore/Presigner(behind the existing interface, no core change) — the S3-compatible path already covers the common cases and is load-proven.commitVerifiedinto a proxy caller.Full detail per change is in
CHANGELOG.md; the design specs are underdocs/specs/2.3.0/.