feat: add Go module proxy collector - #3166
Open
ChrisJr404 wants to merge 1 commit into
Open
Conversation
Adds a collector that pulls module metadata from the Go module proxy (proxy.golang.org) for golang purls. For each module it lists versions via @v/list when the purl has none, then fetches the .info and .mod for each version and emits them as documents. Scoped to collection per the issue; processor/parser wiring for the GO_MOD document type can follow. Signed-off-by: Chris (ChrisJr404) <11917633+ChrisJr404@users.noreply.github.com>
ChrisJr404
force-pushed
the
collector-gomod
branch
from
August 18, 2026 07:17
9570606 to
ba4b9df
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a collector that ingests Go modules straight from the module proxy (proxy.golang.org) so golang purls can be pulled without cloning anything.
For each golang purl it figures out the module path. If the purl carries a version it fetches just that one; if not, it hits
@v/listand expands to every version the proxy knows about. Per version it grabs the.info(version metadata, JSON) and the.mod(the go.mod file) and emits a document for each. Bare module paths work too, which matters because a purl lowercases the namespace and the proxy needs the real casing for modules likegithub.com/BurntSushi/toml— that's why the case-encoding (!before each uppercase letter) lives inescapePathand gets exercised through the bare-path route. Shape follows the deps.dev and pypi collectors:version.UATransportclient, poll/once modes, dedupe across poll cycles, purl parsing viapackageurl-go.I added a
GO_MODdocument type constant alongside the existing ones. This PR is scoped to collection like the issue asks; wiring a processor/parser for the new type is a sensible follow-up.Tests spin up an
httptestserver backed by atestdatatree laid out like the proxy's escaped paths, then drive everything throughcollector.Collect. The.infodocs go throughdochelper.DocTreeEqual; go.mod is plain text so those are compared on raw bytes since the tree helper normalizes blobs as JSON. Cases cover a pinned version, version expansion via@v/list, the uppercase case-encoding, bare paths, dedupe, non-golang purls getting skipped, and a missing module emitting nothing.escapePathalso has its own unit test.go build ./...,go vet,gofmt, andgo test ./pkg/handler/collector/gomod/...all pass locally.Fixes #215