Summary
On-disk API caches under $JENKINS_HOME/org.jenkinsci.plugins.github_branch_source.GitHubSCMProbe.cache accumulate without bound when a multibranch/SCM source authenticates with a rotating token supplied as a standard UsernamePasswordCredentials (rather than the plugin's native GitHubAppCredentials). This is the same disk/inode exhaustion described in JENKINS-63711, but for a credential type the JENKINS-63711 fix (#341) deliberately left out of scope.
Environment
- github-branch-source:
1967.1970.vd86979736546
- Jenkins:
2.568.1
- Auth: GitHub App installation tokens issued externally (in our case via HashiCorp Vault +
vault-plugin-secrets-github), token TTL 1 hour, delivered into Jenkins as a UsernamePasswordCredentials (secret text / username-password), not as GitHubAppCredentials.
Root cause (from current master)
Two behaviors combine:
-
Cache directory name is keyed on the secret value. Connector.getCache() names the cache subdirectory as SHA-256(apiUrl "::" username "::" authHash), where authHash is a digest of the credential secret. A token that rotates hourly therefore produces a new cache directory every rotation, per repo.
-
Automatic cleanup is gated on the concrete credential class. UnusedConnectionDestroyer deletes a cache folder only when cleanupCacheFolder == true, which is set exclusively via:
new GitHubConnection(gb.build(), cache, credentials instanceof GitHubAppCredentials)
For a UsernamePasswordCredentials this is false, so the hourly orphaned directories are never reclaimed.
Net effect: one new, never-deleted cache directory per repo per hour → unbounded directory/inode growth. PR #341 explicitly scoped its cleanup to GitHubAppCredentials ("Behavior for non-GitHubAppCredential connections is unchanged to preserve the advantages of the existing behavior for standard credentials"), which is reasonable for stable standard credentials but breaks down for externally-issued short-lived tokens — an increasingly common pattern (Vault, IRSA/OIDC brokers, other secret managers).
Impact
Operators in this configuration must run an out-of-band cron job to periodically delete the cache directory (we've maintained one for ~3 years), or disable caching entirely via -Dorg.jenkinsci.plugins.github_branch_source.GitHubSCMSource.cacheSize=0 at the cost of losing conditional-request (ETag/304) rate-limit savings.
Suggested directions (any one would resolve it)
- Size/age-based eviction of the cache root (e.g. LRU trim to
cacheSize, or delete entries idle beyond a threshold) that is independent of credential class — this fixes the general case rather than special-casing one credential type.
- Cleanup based on token rotation, not credential class: treat any connection whose
authHash is no longer referenced as eligible for folder deletion in UnusedConnectionDestroyer.
- Decouple the cache key from the secret where a stable identity is available (e.g. key on username / app-id + apiUrl rather than the rotating token), so rotation reuses one directory instead of spawning new ones.
References
Summary
On-disk API caches under
$JENKINS_HOME/org.jenkinsci.plugins.github_branch_source.GitHubSCMProbe.cacheaccumulate without bound when a multibranch/SCM source authenticates with a rotating token supplied as a standardUsernamePasswordCredentials(rather than the plugin's nativeGitHubAppCredentials). This is the same disk/inode exhaustion described in JENKINS-63711, but for a credential type the JENKINS-63711 fix (#341) deliberately left out of scope.Environment
1967.1970.vd869797365462.568.1vault-plugin-secrets-github), token TTL 1 hour, delivered into Jenkins as aUsernamePasswordCredentials(secret text / username-password), not asGitHubAppCredentials.Root cause (from current
master)Two behaviors combine:
Cache directory name is keyed on the secret value.
Connector.getCache()names the cache subdirectory asSHA-256(apiUrl "::" username "::" authHash), whereauthHashis a digest of the credential secret. A token that rotates hourly therefore produces a new cache directory every rotation, per repo.Automatic cleanup is gated on the concrete credential class.
UnusedConnectionDestroyerdeletes a cache folder only whencleanupCacheFolder == true, which is set exclusively via:For a
UsernamePasswordCredentialsthis isfalse, so the hourly orphaned directories are never reclaimed.Net effect: one new, never-deleted cache directory per repo per hour → unbounded directory/inode growth. PR #341 explicitly scoped its cleanup to
GitHubAppCredentials("Behavior for non-GitHubAppCredential connections is unchanged to preserve the advantages of the existing behavior for standard credentials"), which is reasonable for stable standard credentials but breaks down for externally-issued short-lived tokens — an increasingly common pattern (Vault, IRSA/OIDC brokers, other secret managers).Impact
Operators in this configuration must run an out-of-band cron job to periodically delete the cache directory (we've maintained one for ~3 years), or disable caching entirely via
-Dorg.jenkinsci.plugins.github_branch_source.GitHubSCMSource.cacheSize=0at the cost of losing conditional-request (ETag/304) rate-limit savings.Suggested directions (any one would resolve it)
cacheSize, or delete entries idle beyond a threshold) that is independent of credential class — this fixes the general case rather than special-casing one credential type.authHashis no longer referenced as eligible for folder deletion inUnusedConnectionDestroyer.References