Add version and upstream license links to THIRD_PARTY_NOTICES.md - #1
Draft
abrarshivani wants to merge 2 commits into
Draft
abrarshivani wants to merge 2 commits into
abrarshivani wants to merge 2 commits into
Conversation
THIRD_PARTY_NOTICES.md now carries a Version and a Location column instead of the Dependency column. Location links to the license file in the dependency's own upstream repository, pinned to the version we redistribute: | Package | Version | License | Location | |---------|---------|---------|----------| | `github.com/NVIDIA/go-nvlib/pkg` | v0.12.0 | Apache-2.0 | [LICENSE](https://github.com/NVIDIA/go-nvlib/blob/v0.12.0/LICENSE) | Version was dropped in d28112a on the grounds that the notices identify dependencies and their licenses rather than an exact build. That is reversed here: a notices file that does not say which version it describes cannot be matched to a release, and a link into upstream needs a ref to point at. The churn per bump is one index row and two bullets. Every URL was verified by fetching it and comparing its sha256 against the copy under vendor/. A URL that does not match is never written, so no link is dead and none points at the wrong license. 74 URLs across the shipped dependency set. vendor/ gives the module, the version and the license file names for free, but not the upstream repository: k8s.io/api lives at github.com/kubernetes/api, sigs.k8s.io/yaml at github.com/kubernetes-sigs/yaml. Two committed maps carry that instead, both machine-generated. scripts/module-repos.tsv maps module to repository, resolved from the Go module proxy's Origin, then the go-import meta tag that go get itself uses, then the github.com/<org>/<repo> path shape. It is keyed by module and not by version, so a bump does not invalidate it. It covers all 234 vendored modules, not just the shipped ones, so adding a dependency rarely requires re-running it. scripts/license-urls.tsv maps module, version and license path to a verified URL. A row is written only when the bytes at that URL hash identically to the vendored copy. Probing for a 200 is not enough: it cannot tell a correct link from one that returns 200 for the wrong license. Both are produced out of band by 'make third-party-notices-repos' and 'make third-party-notices-urls', which need network. 'make third-party-notices' reads them offline, so 'make check-third-party-notices' stays hermetic. Scope is unchanged. The verifier reuses the generator's own collection, so it covers what go-licenses attributes to ./cmd/...: 56 packages out of 234 vendored modules. The rest are golangci-lint and its plugin tree, vendored for linting and never redistributed. License files are now enumerated from vendor/ rather than from the go-licenses save output, because that output keeps only the one file it classifies as the license per package and drops the rest. That recovers nineteen files the document previously omitted, including nine PATENTS files, four AUTHORS files and the LICENSE.libyaml that sigs.k8s.io/yaml/goyaml.v2 ships alongside its Apache-2.0 LICENSE. scripts/license-overrides.tsv corrects the License column where go-licenses under-reports it. gopkg.in/yaml.v3 ships one LICENSE holding a full-text MIT section plus a short-form Apache-2.0 grant, and sigs.k8s.io/yaml/goyaml.v2 ships LICENSE (Apache-2.0) alongside LICENSE.libyaml (MIT); go-licenses reports a single identifier for each, so both read Apache-2.0 / MIT from the override. Each was confirmed by reading the vendored file rather than by scanning license text, because scanning cannot tell BSD-2-Clause from BSD-3-Clause and a wrong addition is worse than an omission. Generation fails if an override names a package no longer in the index, so the file cannot rot unnoticed. third-party-notices-links.yaml re-verifies every URL weekly. Links are proven correct when written, but upstream can retag or archive a repository afterwards and no offline gate can see that. A version change now needs two commands, because a verified URL contains the version: make third-party-notices-urls # network make third-party-notices # offline Dependabot cannot do the first on its own; its bump job needs wiring, or a human runs it. That is the direct cost of requiring every link to be verified rather than derived. 'make test-tools' runs the new bash suites, 67 assertions across two files. This repo has no 'make check' aggregate, so it runs as a step in the existing third-party-notices workflow alongside the staleness check. Transient failures fetching a license blob are retried. go.googlesource.com returns 503 and 429 under the per-file loop this drives, and the fail-closed gate would otherwise treat a rate-limited response as link rot; the weekly link check drives the same loop. A 404 still fails on the first request, so a real miss costs one request per candidate. fetch_retry moves into license-url-lib.sh as http_fetch_to_file so both resolvers share one retry and status policy, and it writes to a file because command substitution strips the trailing newline that a license file's sha256 depends on. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
abrarshivani
force-pushed
the
tpn-version-location
branch
from
August 27, 2026 19:37
55629be to
6602611
Compare
The image copies a prebuilt gpu-feature-discovery in from the k8s-device-plugin image, and the header said its modules were not covered here. They are redistributed all the same: the binary ships in the image whoever built it, and the third-party code linked into it ships with it. Nothing in this repo records what that binary links, so the binary is the source of truth. Go writes every dependency into the build info at link time, which is what actually ships rather than what a tag implies. go-licenses classifies packages rather than modules, so the package set still has to be resolved from source, and the two are then required to agree — a rebuilt or patched image would otherwise be attributed to the wrong sources without a word. The binary records its own module version as "(devel)", which resolves to nothing, so the version comes from the image tag instead. That mapping is a convention nothing enforces, which is exactly what the dependency cross-check covers: the resolved set has to equal what the binary records, or the run stops. Runtime and bundled entries merge into one index. The two trees are a build-time detail; what ships is one filesystem, so a module both binaries link at different versions is two honest rows rather than a second table the reader has to reconcile. The source tree moves into the row as a sixth field, since it is now a property of the row rather than of the table, and the resolver, verifier and both emit functions read it from there. 56 rows become 91: 13 modules the document did not mention at all, and 22 packages that ship at two versions and now say so. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
VersionandLocationcolumns toTHIRD_PARTY_NOTICES.md, replacing theDependencycolumn.
Locationlinks to the license file in the dependency's own upstream repository, pinnedto the version we redistribute:
github.com/NVIDIA/go-nvlib/pkgThis is the same change as NVIDIA/gpu-operator, applied here.
Version was dropped in d28112a ("Remove Go module versions from rendered TPN document") on the
grounds that the notices identify dependencies and their licenses rather than an exact build.
That is reversed here, and I want to flag it rather than bury it. Two reasons: a notices file
that does not say which version it describes cannot be matched to a release, and a link into
upstream needs a ref to point at. The churn is real but small, one index row and two bullets per
bump.
Every URL in the file was verified by fetching it and comparing its sha256 against the copy under
vendor/. A URL that does not match is never written, so there are no dead links and none pointat the wrong license. 74 URLs.
That includes the secondary license files a module ships alongside its main one, which are easy
to lose: nine
golang.org/x/*modules carryPATENTS, four modules carryAUTHORS, andsigs.k8s.io/yaml/goyaml.v2carriesLICENSE.libyaml(MIT, for the embedded libyaml port) ontop of its Apache-2.0
LICENSE. Nineteen license files that were previously dropped are nowreproduced.
What to review
Hand-written:
scripts/verify-license-urls.shscripts/generate-third-party-notices_test.shscripts/resolve-module-repos.shscripts/license-url-lib.shscripts/license-url-lib_test.sh.github/workflows/third-party-notices-links.yamlscripts/test-helpers.shscripts/generate-third-party-notices.shscripts/license-overrides.tsvMakefile.github/workflows/third-party-notices-check.yamlGenerated, do not read:
THIRD_PARTY_NOTICES.md,scripts/license-urls.tsv(79),scripts/module-repos.tsv(238).scripts/verify-license-urls.shis the one to read. Everything else supports it.How it works
vendor/gives the module, the version and the license file names for free, but it does notrecord the upstream repository.
k8s.io/apiactually lives atgithub.com/kubernetes/api,sigs.k8s.io/yamlatgithub.com/kubernetes-sigs/yaml. Scraping that out of vendored sources iswrong more often than right, so two committed maps carry it instead, both machine-generated:
scripts/module-repos.tsvmaps module to repository. Resolution is the Go module proxy'sOrigin, then thego-importmeta tag thatgo getitself uses, then thegithub.com/<org>/<repo>path shape. Nothing is hand-written. It is keyed by module and not byversion, so a bump does not invalidate it. It covers all 234 vendored modules rather than only
the shipped ones, so adding a dependency rarely requires re-running it.
scripts/license-urls.tsvmaps module, version and license path to a verified URL. A row iswritten only when the bytes at that URL hash identically to the vendored copy. Probing for a 200
is not enough: it cannot tell a correct link from one that returns 200 for the wrong license.
Both are produced out of band by
make third-party-notices-reposandmake third-party-notices-urls, which need network.make third-party-noticesreads them offline, somake check-third-party-noticesstays hermetic and cannot flap on a proxy that withholdsOrigin.Scope is unchanged. The verifier reuses the generator's own collection, so it covers what
go-licensesattributes to./cmd/...: 56 packages out of 234 vendored modules. The rest aregolangci-lint and its plugin tree, vendored for linting and never redistributed.
License files are now enumerated from
vendor/rather than from thego-licenses saveoutput,because that output keeps only the one file it classifies as the license per package and drops
the rest.
scripts/license-overrides.tsvcorrects the License column where go-licenses under-reports it.Two modules here ship a file holding more than one license:
gopkg.in/yaml.v3carries afull-text MIT section plus a short-form Apache-2.0 grant in one
LICENSE, andsigs.k8s.io/yaml/goyaml.v2carriesLICENSE(Apache-2.0) alongsideLICENSE.libyaml(MIT).go-licenses classifies each as a single license, so both read
Apache-2.0 / MITfrom theoverride instead. Each was confirmed by reading the vendored file by eye rather than by scanning
license text, because scanning cannot tell BSD-2-Clause from BSD-3-Clause and a wrong addition to
this file is worse than an omission. Generation fails if an override names a package that is no
longer in the index, so the file cannot rot unnoticed.
.github/workflows/third-party-notices-links.yamlre-verifies every URL weekly. Links are provencorrect when written, but upstream can retag or archive a repository afterwards and no offline
gate can see that.
Note for dependency bumps
A version change now needs two commands, because a verified URL contains the version:
Dependabot cannot do the first on its own. Its bump job needs wiring, or a human runs it. This is
the direct cost of requiring every link to be verified rather than derived.
Testing
make test-toolsruns the new bash suites, 54 assertions across two files. This repo has nomake checkaggregate, so it runs as a step in the existing Third-Party Notices workflowalongside the staleness check.
Verified by hand:
HEADor branch refsvendor/modules.txtgo-nvlibrows fromscripts/license-urls.tsvmakes
make third-party-noticesexit non-zero naming that module, and alicense-overrides.tsvrow for a package not in the index does the same