feat(git-providers): first-class repositories behind a provider interface - #6939
feat(git-providers): first-class repositories behind a provider interface#6939viktormarinho wants to merge 8 commits into
Conversation
Deployment setup, step by stepDecision taken: reuse the same GitHub App that backs Nothing here is required to merge. With none of these variables set, the connect routes answer 0. Before you touch anythingTwo fields on a GitHub App are single-valued and shared with
Private keys and client secrets are additive: generating a new one does not revoke existing ones. Never delete the key 1. Open the App's settingsThe App is owned by the organization, so its settings live at: If you do not know the slug, find the App under 2. Collect the four values on that page
Then, further down the same page:
3. Register Studio's URLs on the AppReplace
4. Set the variablesSelf-hosted / docker compose — see GITHUB_APP_ID=
GITHUB_APP_PRIVATE_KEY=
GITHUB_APP_CLIENT_ID=
GITHUB_APP_CLIENT_SECRET=
GITHUB_APP_SLUG=Helm — the keys exist in secret:
GITHUB_APP_ID: "..."
GITHUB_APP_PRIVATE_KEY: |
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
GITHUB_APP_CLIENT_ID: "..."
GITHUB_APP_CLIENT_SECRET: "..."
GITHUB_APP_SLUG: "..."Unrelated but easy to confuse: 5. Verify
Nothing migrates on its own — an org keeps using its GitLab, when you get theregitlab.com needs an OAuth application. For an org-wide integration create it on the group (
GITLAB_OAUTH_HOST=gitlab.com
GITLAB_OAUTH_CLIENT_ID=
GITLAB_OAUTH_CLIENT_SECRET=Self-managed GitLab needs no deployment configuration at all. An OAuth application is per instance, so instead the customer connects with an access token in Settings → Repositories → Connect with a token, entering their host (e.g. One more thingThe sandbox image gained |
af800b1 to
a639a4a
Compare
Ran the flow locally against a live GitLab accountBrought Studio up locally ( 1. A long-lived access token could never start a sandbox. 2. "Add repository" was disabled with no connected account, while the copy beside it offered to link a public repository by URL — which needs no account. The button is enabled; the now-orphaned 3. The token dialog asked for the wrong scopes ( What was exercisedConnect by token → the account row renders ( The error path was checked too: an invalid token surfaces GitLab's own message ( Still not covered locallyThe sandbox pod itself, which needs the k8s provider — the clone credential it consumes is what was verified above, and the daemon's side is covered by its Go tests. And the GitHub App installation mint, which needs a registered App; its JWT signing is unit-tested. All gates re-run after the fixes: typecheck, |
9c3ed73 to
281954e
Compare
…face Repositories become an org entity instead of an `owner/name` pair scattered across connection metadata and denormalized column pairs, and Studio talks to GitHub and GitLab over their REST APIs directly rather than through an MCP round-trip. - `git_provider_accounts` holds the credential; its `type` selects the provider client and `auth_kind` how it authenticates (GitHub App installation, OAuth, or an access token). `repositories` is keyed by (org, host, path), so a GitLab namespace of any depth fits where `owner/name` did not. - `GitProviderClient` is the seam: one implementation per provider, resolved from the account row. Studio now owns the GitHub App JWT and mints installation tokens itself; GitLab uses OAuth with refresh or a token. - The OAuth refresh helpers are generic over an `OAuthGrantStore`, so one refresh path serves both `downstream_tokens` and the new account store. - Sandbox start, credential re-mint and push refresh use a repository's Studio-owned credentials when it has them, and fall back to the existing `mcp-github` path otherwise. The daemon derives its CLI environment from the clone URL's userinfo (`gh` or `glab`) and no longer compares against a literal `github.com` before refusing an uncredentialed push. Every schema change is additive: consumers gain a nullable `repository_id` alongside their existing columns, backfilled from the current metadata. Task board, the web PR panel and the reports service still take the legacy path; they follow on top of this.
…ication Both found by exercising the provider clients against the live APIs rather than a stub. - `listRepos` asked gitlab.com for `membership=true` ordered by `last_activity_at`, which answers 500 after ~15s, reproducibly (`updated_at` and `name` too; `owned=true` with the same ordering is fine). Order by `id` instead, and take the `simple` representation — it carries every field the summary needs, contrary to the comment that justified omitting it. - `glab` cannot authenticate from the environment for an OAuth access token: it sends an env token as `PRIVATE-TOKEN`, which GitLab rejects for one, and every documented variable behaves the same. The daemon now writes glab's config file (0600, `is_oauth2: true` so the token goes out as a bearer, which is accepted for personal and project access tokens as well) before each run, and removes it for a non-GitLab remote so a pod that switches repositories cannot leave another provider's token behind. Also adds the e2e spec for the tool surface: a repository is org-scoped and keyed by (host, path) case-insensitively, GitLab subgroups survive a merge-request URL, and one org can neither list nor delete another's row.
Repositories were only reachable through the MCP tools; this gives them a surface, and points the two existing repo consumers at it. - **Settings → Repositories** lists the org's provider accounts and linked repositories. Connecting through OAuth needs deployment credentials, so those buttons appear only when `GIT_PROVIDER_CAPABILITIES` reports them; connecting GitLab with an access token needs nothing and is always offered, which is the path a self-managed instance takes. Adding a repository either searches an account's projects or takes a pasted URL. - **The repo picker** now reads the new model whenever the org has a serviceable account: it lists already-linked repositories first, links a new one through `REPOSITORY_LINK`, and stamps `metadata.githubRepo.repositoryId` on the agent it creates. An org still on `mcp-github` keeps the old flow untouched, so nothing changes until an account is connected. - **Repo → volume sync** takes a `repositoryId` as its source instead of a repo-scoped connection, and fetches the archive through the provider client (`archiveTarball`), which makes a GitLab-hosted sync work. `connectionId` keeps working for configs that predate the model; migration 202 relaxes its NOT NULL and replaces it with "at least one source".
Driving the real Settings → Repositories page against gitlab.com surfaced three defects the unit and e2e suites could not. - `SANDBOX_START` always asks for a freshly-minted credential, and the token source passed that straight through as `force`. A personal or project access token has no refresh token and no expiry, so forcing it took `getValidDownstreamAccessToken`'s `expired_without_refresh` branch and yielded null: every sandbox backed by a GitLab token account would have failed to clone. Force now applies only to a grant that can be refreshed. - The "add repository" button was disabled without a connected account, while the copy beside it offered to link a public repository by URL — which needs no account at all. - The token dialog asked for `read_api` and `read_repository`, neither of which can push a branch or open a merge request. It asks for `api`. Verified against a live GitLab account: connect by token, search the account's projects, link one, and clone the private repository with the credential the provider account minted.
…igration The NOT NULL drop and the "at least one source" check belong with the migration that introduces `repository_id` in the first place — they only exist because a repository-backed sync names no connection. One migration now carries the whole model, and its down() reverses it in the order the constraint requires.
…ndbox `TASK_ADD_REPO` and the secondary-checkout path only ever knew `mcp-github` connections, so a linked GitLab repository was invisible to the agent — and a repository-backed secondary was skipped silently, without an error, because the loop required a `connectionId`. - The tool lists repositories and legacy connections together, deduped by (host, path) with the repository winning, so an org mid-migration sees each repository once through the credential Studio can actually mint. Its identifier is now an opaque `id`; `connectionId` stays as a deprecated alias. - Secondary checkouts resolve through either model, in `SANDBOX_START` and in the tool's own config push. Mixing is the point: the daemon clones each checkout from its own credentialed URL, so a GitHub primary alongside a GitLab secondary is just two independent clones. - The in-pod CLI login is provider-aware (`gh` hosts.yml or glab's config, keyed by the remote's host) and now runs *in the checkout being added*. It used to run in the daemon's cwd — the primary — so a GitLab secondary would have configured `gh` from the primary's remote and left `glab` unauthenticated. The two providers keep separate config files, so a mixed-provider sandbox ends up with both working. The agent-facing copy no longer promises `gh` specifically. `mergeRepoChoices` and `cliAuthCommand` are pure and unit-tested, including the nested-namespace owner split and the two-hosts-same-path case.
…h providers `TASK_ADD_REPO` was converted last commit but its two siblings were not, so which repos an agent could reach depended on which entry point it came in through — `load_repo` (Decopilot chat) and the claude-code task-run dispatch still listed only `mcp-github` connections. The selection logic moves to `git-providers/repo-choices.ts` and all three share it: repositories from any provider, plus legacy connections, deduped by (host, path) with the repository winning. `load_repo` takes the same opaque `id`, keys its sandbox off it, and writes `repositoryId` on the thread binding so `SANDBOX_START` resolves credentials without minting here. Two behaviours worth naming: - `orgSharedFirst` is new and prevents a regression: `pickSoleTaskRepo` used to prefer the org-shared `mcp-github` connection when one repo had two, because the per-agent child dies with its agent. Dedup keeps the first entry, so the org-shared one has to sort ahead. - A repository whose provider is not GitHub no longer opens on Site Editor, which reads the decofile over GitHub's Git Data API and cannot load. Those projects open on Chat — the coding agent, which does work — until the editor speaks both providers. The agent-facing prompts stop claiming `gh` is the CLI. They still say `gh pr create`, which is wrong on a GitLab run and is the next thing to fix.
…providers The run prompt hardcoded `gh pr create` and "pull request", which is simply wrong instructions on a GitLab repository — the agent would run a command the checkout has no CLI for. The provider now travels on the repo choice, and the instructions that name a command take their vocabulary from it (`providerCli` in `@decocms/shared/git-providers`: binary, create/checkout command, and what the provider calls a proposed change). Naming only the primary's CLI would be its own bug, though: `TASK_ADD_REPO` accumulates checkouts and they can be on different hosts, so a run may hold a GitHub repository and a GitLab one at once. Every prompt therefore also states the rule — a checkout is authenticated for ITS OWN host, `git remote get-url origin` settles which — and the reviewer prompt says the same.
281954e to
6ff2f02
Compare
1 participant
Repositories become an org entity instead of an
owner/namepair scattered across connection metadata and denormalized column pairs, and Studio talks to GitHub and GitLab over their REST APIs directly instead of through an MCP round-trip. This is the foundation for onboarding GitLab customers; the task board, the PR panel and the reports service still take the legacy path and follow on top of this.The model
git_provider_accountsholds the credential — itstypeselects the provider client, itsauth_kindhow it authenticates (GitHub App installation, OAuth, or an access token).repositoriesis keyed by(org, host, path)case-insensitively, so a GitLab namespace of any depth fits whereowner/namedid not.GitProviderClientis the seam: one implementation per provider, resolved from the account row. Studio now owns the GitHub App JWT and mints installation tokens itself; GitLab uses OAuth with refresh, or a personal/project access token. The OAuth refresh helpers became generic over anOAuthGrantStore, so one refresh path serves bothdownstream_tokensand the new account store — no new credential storage was invented.What consumes it
metadata.githubRepo.repositoryIdon the agent it creates. An org still onmcp-githubkeeps the old flow untouched.repositoryIdand fetches througharchiveTarball, which makes a GitLab-hosted sync work.The daemon derives its CLI environment from the clone URL's userinfo (
ghorglab) and no longer compares against a literalgithub.combefore refusing an uncredentialed push.glabis now in the sandbox image.Every schema change is additive: consumers gain a nullable
repository_idalongside their existing columns, backfilled from current metadata. Migration 202 relaxesorg_repo_sync.connection_id's NOT NULL and replaces it with "at least one source".Two bugs the live APIs caught
Exercising the clients against real GitHub and GitLab, rather than a stub, found two things a stub could not:
listReposasked gitlab.com formembership=trueordered bylast_activity_at, which answers 500 after ~15s, reproducibly (updated_atandnametoo;owned=truewith the same ordering is fine). It now orders byidand takes thesimplerepresentation — which does carry every field the summary needs, contrary to the comment that justified omitting it.glabcannot authenticate from the environment for an OAuth token: it sends an env token asPRIVATE-TOKEN, which GitLab rejects for one, and every documented variable (GITLAB_TOKEN,GITLAB_ACCESS_TOKEN,OAUTH_TOKEN) behaves the same. The daemon now writes glab's config file (0600,is_oauth2: trueso the token goes out as a bearer), and removes it for a non-GitLab remote so a pod that switches repositories cannot leave another provider's token behind.Testing
RepoRefparsing including GitLab subgroups and/-/sub-resources, the daemon's CLI-env and glab-config builders, and the push guard.packages/e2e/tests/git-providers.spec.ts, new): a repository is org-scoped and keyed case-insensitively, a GitLab project nested in subgroups survives a merge-request URL, invalid URLs are refused, link/list/delete round-trips, and one org can neither list nor delete another's row — the cross-tenant test sets an active org explicitly, since a signup-only session leavesactiveOrganizationIdNULL and would pass even with the gate broken.org-repo-sync.spec.tsgains repository-backed coverage.getRepoon a three-level namespace,readFile, 404s,listRepos, clone withoauth2:<token>@, commit, push,ls-remote --symref, and a merge request opened with the exact config the daemon writes. On GitHub, identity, repo reads, 404s and search, plus confirmation thatghauthenticates fromGH_TOKENalone. Test resources were deleted afterwards.downand re-upverified.Not verified end to end: the GitHub App installation-token mint, which needs a registered App (browser flow). Its JWT signing is unit-tested.
Deploying
Dormant until configured — with no credentials the connect routes answer 503 and existing
mcp-githubconnections keep working. Seeselfhost/examples/docker-compose/.env.examplefor the full list (GITHUB_APP_*,GITLAB_OAUTH_*); the helm secret template carries the same keys.The decision that unblocks the rest: whether the GitHub App backing
deco/mcp-githubmoves to Studio (itsinstallation_ids stay valid and migration is transparent) or a new App is registered (every org must reinstall).Summary by cubic
Repositories become an org-scoped entity behind a provider interface —
git_provider_accountsholds credentials andrepositoriesis keyed by (org, host, path) — and Studio talks to GitHub and GitLab over their REST APIs directly instead of an MCP round-trip. This is the foundation for onboarding GitLab; the PR panel and reports service still use the legacy path.Changes
OAuthGrantStorepath.mcp-github.load_repo, task-board dispatch,TASK_ADD_REPO) list repositories and legacy connections together, deduped by (host, path) with the repository winning, so a GitHub primary and a GitLab secondary coexist in one sandbox.repositoryIdand fetches viaarchiveTarball, making GitLab syncs work.ghandglabboth work; the daemon derives CLI env from the clone URL and writes glab's config (0600) for OAuth tokens.listReposnow orders byidwith thesimplerepresentation (the prior query 500s); force-refresh only applies to refreshable grants, and the token dialog asks forapi.connection_idNOT NULL for "at least one source".Deploying
mcp-githubkeeps working.GITHUB_APP_*/GITLAB_OAUTH_*env vars (see.env.exampleand the helm secret template).deco/mcp-githubmoves to Studio (installations stay valid) or a new App is registered (orgs must reinstall).Written for commit 6ff2f02. Summary will update on new commits.