Skip to content

feat(git-providers): the CMS reads and writes through a provider interface - #7015

Merged
viktormarinho merged 10 commits into
mainfrom
t3code/git-providers-repo-content
Sep 8, 2026
Merged

feat(git-providers): the CMS reads and writes through a provider interface#7015
viktormarinho merged 10 commits into
mainfrom
t3code/git-providers-repo-content

Conversation

@viktormarinho

@viktormarinho viktormarinho commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Stacked on #6939 — review that first; this PR's diff is only the CMS/decofile layer.

The Site Editor was GitHub-only by construction, not by omission. GitDataClient was shaped like GitHub's Git Data API — create a blob, then a tree, then a commit, then move a ref — and GitLab exposes no such plumbing at all, so an interface in that shape has no second implementation. That is what blocked a GitLab storefront from using the CMS.

RepoContentClient states the write side as the intent instead: put these files on this branch, atomically, unless the branch moved. GitHub does it in four calls, GitLab in one; neither leaks through. The read side needed no lift — both providers address tree entries and blobs by object sha, and both accept a commit sha wherever a tree-ish is asked for.

Net: +449 / −1274. decofile/github-git-data.ts and client-for-repo.ts are gone.

The three properties the plumbing carried implicitly

Losing one of these silently was the real risk of this refactor, so each is now an explicit part of the contract:

what it is for GitHub GitLab
expectedHead the commit coalescer's multi-replica safety non-forced ref update, 422 → RepoWriteConflict no branch-level CAS: reads the head and guards every changed file with last_commit_id, because the head check alone leaves a window
rewriteFrom the squash-rebase commit first, force-move the ref last — a crash cannot strand the branch one forced call (start_sha + force); falls back to replacing the branch only when it is protected, which is the one case with a window
copyFromRef "this path should have the content it has at that ref" points the new tree at the blob that already exists — zero uploads, which is what the replay used to do by hand reads and rewrites it

FileChange also carries an optional mode, so the exec bit survives a discard.

Measured, not assumed

Everything above about GitLab came from probing gitlab.com directly (throwaway projects, deleted after):

  • multi-file atomic commit via actions[] → 201, one commit, one parent
  • optimistic locking is per file: a stale last_commit_id → 400 {"message":"The file has changed since you started editing it: a/one.json"}
  • committing onto an existing branch from an earlier start_sha → 400 "A branch called 'main' already exists"
  • force: true lifts exactly that refusal — 201, parent = the base sha, prior content gone — except on a protected branch ("not allowed to force push"), and main is protected by default
  • execute_filemode: true → mode 100755
  • tree rows carry the object sha as id, blobs/:sha/raw serves content, and a commit sha works as a ref
  • there is no POST /repository/merge (404): merging a branch means creating an MR and merging it

Testing

  • Unit: 219 in content/ + decofile/ — tree walking, the change→tree-entry mapping with modes and copyFromRef, the GitLab action builder, and each conflict classifier.
  • E2E: 253 pass. decofile-api, cms-publish-stages, cms-publish-surface and fast-preview-git-sync (35 specs) run green against the new interface with the GitHub stub unmodified. The one failure is mcp-proxy-roundtrip, which fails identically on a clean main — pre-existing, unrelated.
  • The squash ordering was verified over real HTTP against the stub: call order is POST /git/trees → POST /git/commits → GET /branches → PATCH /git/refs, i.e. the commit strictly precedes the ref move, and a clean sync issues zero POST /git/blobs.

Known limitations

  • GitLab reads block-by-block. Its tree listing carries no blob size, and the decofile cold read refuses the tarball fast path when a size is missing (it pre-validates the aggregate before downloading). Correct, but slower on a large storefront. Fixing it means enforcing the cap during extraction instead of before — a memory-safety guard I did not want to weaken in this PR.
  • A clean sync over 300 files still 409s. GitHub truncates a compare's files at 300 with no total, so a copyFromRef replay built from it could be partial — and a partial replay would silently revert the paths it never saw. The guard stays until the interface can express "the resulting tree is exactly this ref's".
  • mergeBranches on GitLab leaves a merge request behind, because no direct branch-merge endpoint exists.

Summary by cubic

Replaces the GitHub-shaped GitDataClient with a provider-neutral RepoContentClient, so the CMS can read and write through GitHub or GitLab. Writes now express atomic branch intent instead of exposing GitHub’s blob/tree/commit/ref sequence; GitHub still uses four calls, while GitLab uses one.

Implementation

  • Makes expectedHead, rewriteFrom, copyFromRef, and file modes explicit in the write contract.
  • Guards GitLab changes with per-file last_commit_id because GitLab has no branch-level compare-and-swap.
  • Routes repository resolution, error handling, reads, writes, rebases, discards, and fast preview through the provider client.
  • Removes the old GitHub-only client and the stale duplicate migration left by the merge.

Validation and limitations

  • Unit and E2E suites pass; the GitHub E2E stub remains unchanged.
  • GitLab reads block-by-block because tree entries do not include blob sizes.
  • GitHub clean syncs over 300 files still return 409 because compare results are truncated.
  • GitLab mergeBranches leaves a merge request behind.

Written for commit 7ea08d7. Summary will update on new commits.

Review in cubic

…face

Repositories become an org entity instead of an `owner/name` pair scattered
across connection metadata and denormalized column pairs, and Studio talks to
GitHub and GitLab over their REST APIs directly rather than through an MCP
round-trip.

- `git_provider_accounts` holds the credential; its `type` selects the provider
  client and `auth_kind` how it authenticates (GitHub App installation, OAuth,
  or an access token). `repositories` is keyed by (org, host, path), so a
  GitLab namespace of any depth fits where `owner/name` did not.
- `GitProviderClient` is the seam: one implementation per provider, resolved
  from the account row. Studio now owns the GitHub App JWT and mints
  installation tokens itself; GitLab uses OAuth with refresh or a token.
- The OAuth refresh helpers are generic over an `OAuthGrantStore`, so one
  refresh path serves both `downstream_tokens` and the new account store.
- Sandbox start, credential re-mint and push refresh use a repository's
  Studio-owned credentials when it has them, and fall back to the existing
  `mcp-github` path otherwise. The daemon derives its CLI environment from the
  clone URL's userinfo (`gh` or `glab`) and no longer compares against a
  literal `github.com` before refusing an uncredentialed push.

Every schema change is additive: consumers gain a nullable `repository_id`
alongside their existing columns, backfilled from the current metadata.

Task board, the web PR panel and the reports service still take the legacy
path; they follow on top of this.
…ication

Both found by exercising the provider clients against the live APIs rather
than a stub.

- `listRepos` asked gitlab.com for `membership=true` ordered by
  `last_activity_at`, which answers 500 after ~15s, reproducibly (`updated_at`
  and `name` too; `owned=true` with the same ordering is fine). Order by `id`
  instead, and take the `simple` representation — it carries every field the
  summary needs, contrary to the comment that justified omitting it.
- `glab` cannot authenticate from the environment for an OAuth access token:
  it sends an env token as `PRIVATE-TOKEN`, which GitLab rejects for one, and
  every documented variable behaves the same. The daemon now writes glab's
  config file (0600, `is_oauth2: true` so the token goes out as a bearer,
  which is accepted for personal and project access tokens as well) before
  each run, and removes it for a non-GitLab remote so a pod that switches
  repositories cannot leave another provider's token behind.

Also adds the e2e spec for the tool surface: a repository is org-scoped and
keyed by (host, path) case-insensitively, GitLab subgroups survive a
merge-request URL, and one org can neither list nor delete another's row.
Repositories were only reachable through the MCP tools; this gives them a
surface, and points the two existing repo consumers at it.

- **Settings → Repositories** lists the org's provider accounts and linked
  repositories. Connecting through OAuth needs deployment credentials, so
  those buttons appear only when `GIT_PROVIDER_CAPABILITIES` reports them;
  connecting GitLab with an access token needs nothing and is always offered,
  which is the path a self-managed instance takes. Adding a repository either
  searches an account's projects or takes a pasted URL.
- **The repo picker** now reads the new model whenever the org has a
  serviceable account: it lists already-linked repositories first, links a new
  one through `REPOSITORY_LINK`, and stamps `metadata.githubRepo.repositoryId`
  on the agent it creates. An org still on `mcp-github` keeps the old flow
  untouched, so nothing changes until an account is connected.
- **Repo → volume sync** takes a `repositoryId` as its source instead of a
  repo-scoped connection, and fetches the archive through the provider client
  (`archiveTarball`), which makes a GitLab-hosted sync work. `connectionId`
  keeps working for configs that predate the model; migration 202 relaxes its
  NOT NULL and replaces it with "at least one source".
Driving the real Settings → Repositories page against gitlab.com surfaced
three defects the unit and e2e suites could not.

- `SANDBOX_START` always asks for a freshly-minted credential, and the token
  source passed that straight through as `force`. A personal or project access
  token has no refresh token and no expiry, so forcing it took
  `getValidDownstreamAccessToken`'s `expired_without_refresh` branch and
  yielded null: every sandbox backed by a GitLab token account would have
  failed to clone. Force now applies only to a grant that can be refreshed.
- The "add repository" button was disabled without a connected account, while
  the copy beside it offered to link a public repository by URL — which needs
  no account at all.
- The token dialog asked for `read_api` and `read_repository`, neither of
  which can push a branch or open a merge request. It asks for `api`.

Verified against a live GitLab account: connect by token, search the account's
projects, link one, and clone the private repository with the credential the
provider account minted.
…igration

The NOT NULL drop and the "at least one source" check belong with the
migration that introduces `repository_id` in the first place — they only
exist because a repository-backed sync names no connection. One migration
now carries the whole model, and its down() reverses it in the order the
constraint requires.
…ndbox

`TASK_ADD_REPO` and the secondary-checkout path only ever knew `mcp-github`
connections, so a linked GitLab repository was invisible to the agent — and a
repository-backed secondary was skipped silently, without an error, because
the loop required a `connectionId`.

- The tool lists repositories and legacy connections together, deduped by
  (host, path) with the repository winning, so an org mid-migration sees each
  repository once through the credential Studio can actually mint. Its
  identifier is now an opaque `id`; `connectionId` stays as a deprecated alias.
- Secondary checkouts resolve through either model, in `SANDBOX_START` and in
  the tool's own config push. Mixing is the point: the daemon clones each
  checkout from its own credentialed URL, so a GitHub primary alongside a
  GitLab secondary is just two independent clones.
- The in-pod CLI login is provider-aware (`gh` hosts.yml or glab's config,
  keyed by the remote's host) and now runs *in the checkout being added*.
  It used to run in the daemon's cwd — the primary — so a GitLab secondary
  would have configured `gh` from the primary's remote and left `glab`
  unauthenticated. The two providers keep separate config files, so a
  mixed-provider sandbox ends up with both working.

The agent-facing copy no longer promises `gh` specifically. `mergeRepoChoices`
and `cliAuthCommand` are pure and unit-tested, including the nested-namespace
owner split and the two-hosts-same-path case.
…h providers

`TASK_ADD_REPO` was converted last commit but its two siblings were not, so
which repos an agent could reach depended on which entry point it came in
through — `load_repo` (Decopilot chat) and the claude-code task-run dispatch
still listed only `mcp-github` connections.

The selection logic moves to `git-providers/repo-choices.ts` and all three
share it: repositories from any provider, plus legacy connections, deduped by
(host, path) with the repository winning. `load_repo` takes the same opaque
`id`, keys its sandbox off it, and writes `repositoryId` on the thread binding
so `SANDBOX_START` resolves credentials without minting here.

Two behaviours worth naming:

- `orgSharedFirst` is new and prevents a regression: `pickSoleTaskRepo` used to
  prefer the org-shared `mcp-github` connection when one repo had two, because
  the per-agent child dies with its agent. Dedup keeps the first entry, so the
  org-shared one has to sort ahead.
- A repository whose provider is not GitHub no longer opens on Site Editor,
  which reads the decofile over GitHub's Git Data API and cannot load. Those
  projects open on Chat — the coding agent, which does work — until the editor
  speaks both providers.

The agent-facing prompts stop claiming `gh` is the CLI. They still say
`gh pr create`, which is wrong on a GitLab run and is the next thing to fix.
…providers

The run prompt hardcoded `gh pr create` and "pull request", which is simply
wrong instructions on a GitLab repository — the agent would run a command the
checkout has no CLI for.

The provider now travels on the repo choice, and the instructions that name a
command take their vocabulary from it (`providerCli` in
`@decocms/shared/git-providers`: binary, create/checkout command, and what the
provider calls a proposed change).

Naming only the primary's CLI would be its own bug, though: `TASK_ADD_REPO`
accumulates checkouts and they can be on different hosts, so a run may hold a
GitHub repository and a GitLab one at once. Every prompt therefore also states
the rule — a checkout is authenticated for ITS OWN host, `git remote get-url
origin` settles which — and the reviewer prompt says the same.
…rface

`GitDataClient` was shaped like GitHub's Git Data API — create a blob, then a
tree, then a commit, then move a ref. GitLab exposes no such plumbing, so an
interface in that shape has no second implementation; the Site Editor was
GitHub-only by construction.

`RepoContentClient` states the write side as the intent instead: put these
files on this branch, atomically, unless the branch moved. GitHub does it in
four calls, GitLab in one, and neither leaks. The read side needed no lift —
both providers address tree entries and blobs by object sha and accept a
commit sha wherever a tree-ish is asked for.

Three properties the plumbing carried implicitly are now explicit, because
losing them silently was the risk:

- `expectedHead` is the coalescer's multi-replica safety. GitHub gets it from
  a non-forced ref update; GitLab has no branch-level CAS, so it reads the head
  and additionally guards every changed file with `last_commit_id` — both
  layers, because the head check alone leaves a window.
- `rewriteFrom` is the squash. GitHub commits first and force-moves the ref
  last, so a crash cannot strand the branch. GitLab does it in one forced call
  (`start_sha` + `force`), falling back to replacing the branch only when it is
  protected, which is the one case that has a window.
- `copyFromRef` says "this path should have the content it has at that ref".
  GitHub points the new tree at the blob that already exists — no upload, which
  is what the replay used to do by hand; GitLab reads and rewrites it.

`FileChange` carries an optional mode, so the exec bit survives a discard.

Measured against gitlab.com rather than assumed: the atomic multi-file commit,
the per-file conflict (400 "The file has changed since you started editing
it"), the refusal to commit onto an existing branch from an earlier
`start_sha`, that `force` lifts exactly that refusal outside a protected
branch, and `execute_filemode`.

Known limitation: GitLab's tree listing carries no blob size, and the decofile
cold read refuses the tarball path when a size is missing, so GitLab
repositories read block-by-block. Correct, but slower on a large storefront.
Base automatically changed from t3code/assess-github-gitlab-integration to main September 8, 2026 17:24
Three conflicts, all of them the resolution already made on the parent branch
arriving through its squash — this branch changed none of the three files
itself, so main's version is taken wholesale.

The squash also left `201-git-provider-accounts-and-repositories.ts` behind
next to the `204-` it was renamed to: this branch's ancestry carried the old
name, main brought the new one, and a merge keeps both. Byte-identical, and
only 204 is registered, so it was dead weight rather than a second migration —
removed here.
@viktormarinho
viktormarinho merged commit 6b89e90 into main Sep 8, 2026
33 checks passed
@viktormarinho
viktormarinho deleted the t3code/git-providers-repo-content branch September 8, 2026 17:36
decocms Bot pushed a commit that referenced this pull request Sep 8, 2026
PR: #7015 feat(git-providers): the CMS reads and writes through a provider interface
Bump type: minor

- decocms (apps/api/package.json): 4.336.0 -> 4.337.0
- @decocms/native (apps/native/package.json): 4.336.0 -> 4.337.0
- @decocms/e2e (packages/e2e/package.json): 1.66.0 -> 1.67.0

Deploy-Scope: server
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant