feat: add server-side pagination to API endpoints by fabiovincenzi · Pull Request #1464 · finos/git-proxy

fabiovincenzi · 2026-03-20T11:16:19Z

Fixes #1297
Implements server side pagination for /push, /repo and /users endpoints.

Note: The list endpoints (/push, /repo, /users) now return { data, total } instead of a plain array to support pagination metadata.

…ed results

… views

…tionParams

netlify · 2026-03-20T11:16:25Z

✅ Deploy Preview for endearing-brigadeiros-63f9d0 canceled.

Name	Link
🔨 Latest commit	`e737d6e`
🔍 Latest deploy log	https://app.netlify.com/projects/endearing-brigadeiros-63f9d0/deploys/69c69ebc12a5650008c7f81a

codecov · 2026-03-20T11:24:36Z

Codecov Report

❌ Patch coverage is 92.28188% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.73%. Comparing base (4e2eea8) to head (e737d6e).

Files with missing lines	Patch %	Lines
src/service/routes/push.ts	55.00%	9 Missing ⚠️
src/db/file/users.ts	85.00%	3 Missing ⚠️
src/db/mongo/pushes.ts	93.02%	3 Missing ⚠️
src/db/mongo/repo.ts	83.33%	3 Missing ⚠️
src/db/mongo/users.ts	90.00%	3 Missing ⚠️
src/db/file/helper.ts	92.59%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1464      +/-   ##
==========================================
+ Coverage   89.66%   89.73%   +0.06%     
==========================================
  Files          68       68              
  Lines        4869     5048     +179     
  Branches      888      931      +43     
==========================================
+ Hits         4366     4530     +164     
- Misses        485      500      +15     
  Partials       18       18

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

dcoric

Thanks for the solid work on server-side pagination. I have a few concerns - mostly around security and a couple of behavioral issues that should be addressed before merging.

src/service/routes/utils.ts

src/service/routes/repo.ts

src/service/routes/push.ts

src/ui/views/RepoDetails/Components/AddUser.tsx

src/db/mongo/helper.ts

src/ui/views/PushRequests/components/PushesTable.tsx

roskosm1

docs API breaking changes?

src/db/mongo/helper.ts

…g on sensitive fields

…patch routes

… select dropdown

jescalada

Oddly, the repo list sorting doesn't seem to be working right on my end - whenever I change the sorting order, the dropdown selector just resets (and only the default alphabetical order is shown regardless of which one I choose).

Related to that, when I move to another page the selected filter gets reset, which (might) be resetting the filter to alphabetical ascending order.

I don't get any error output on console, BTW 🤷🏼

jescalada · 2026-03-24T13:47:56Z

src/db/file/helper.ts

+      });
+  });
+
+  return Promise.all([dataPromise, countPromise]).then(([data, total]) => ({ data, total }));


I'm wondering if we should change the data variable to something more descriptive... Right now, we end up calling response.data.data to fetch the repos/users/etc. which can trip up people/agents working with the code.

I'd suggest having the name of the entity { users, total }, etc. although we'd need to pass in the entity name to paginatedFind (unless it's already obtainable in the existing function parameters).

Good point, the HTTP routes now return entity-specific keys (pushes, repos, users) instead of data. The data variable inside paginatedFind remains as a generic internal name but not exposed directly.

jescalada · 2026-03-24T13:56:40Z

packages/git-proxy-cli/index.ts

  try {
    const cookies = JSON.parse(fs.readFileSync(GIT_PROXY_COOKIE_FILE, 'utf8'));
-    const { data } = await axios.get<Action[]>(`${baseUrl}/api/v1/push/`, {
+    const response = await axios.get<{ data: Action[]; total: number }>(`${baseUrl}/api/v1/push/`, {


Can use PaginatedResult type here

the /push endpoint now returns pushes instead of data.

jescalada · 2026-03-24T14:12:05Z

src/db/mongo/pushes.ts

  type: 'push',
 };

+const pushProjection = {


Perhaps this could be a constant (and potentially placed somewhere else)? Wondering if there are any other constants from the db module that we could wrap into a /db/constants.ts file.

On another note, I'm wondering why we have been doing manual projection at all - if we want to exclude certain fields, we could do post-processing just like we do with users (toPublicUser).

Fixing this should also help simplify the tests that rely on projection 👍🏼

I'm not sure about the reasons for projections here, filtering at the DB level avoids transferring data that is not needed, such as steps which can be quite large. Post-processing like toPublicUser, would work but at the cost of fetching full documents first. Not sure though if this is the reason

Would it be worth tracking in a separate issue/PR to keep this one focused?

jescalada · 2026-03-24T14:28:34Z

src/service/routes/utils.ts

+  const skip = (page - 1) * limit;
+
+  const pagination: PaginationOptions = { skip, limit };
+  if (req.query['search']) pagination.search = req.query['search'] as string;


Wondering if we could have stronger typing for req here - so we could do the following:

const { search, sortBy, sortOrder } = req.query;

Extending the Request type can be a huge pain, but it's been done before with the user property for the passport strategies. This might be more appropriate for a separate PR if it requires too many extra changes!

Done, introduced a PaginationQuery interface and cast req.query once at the top, allowing destructuring without repeated as string casts. I didn't make the global Request type since query params vary per route.

fabiovincenzi · 2026-03-25T10:29:12Z

Oddly, the repo list sorting doesn't seem to be working right on my end - whenever I change the sorting order, the dropdown selector just resets (and only the default alphabetical order is shown regardless of which one I choose).

Related to that, when I move to another page the selected filter gets reset, which (might) be resetting the filter to alphabetical ascending order.

I don't get any error output on console, BTW 🤷🏼

@jescalada Good catch. I tried to replicate the client side sorting behaviour we had before without verifying that lastModified and dateCreated actually exist on the Repo model and they don't, so the sort was always falling back to the default (name). On the old client-side implementation it appeared to work there because the stable sort preserved DB insertion order, which happened to differ from alphabetical. To fix this properly we'd need to add
these fields to the model and populate them, but maybe this is out of scope for this PR? Maybe removing Date Modified and Date Created from the dropdown and open a separate issue to implement them correctly?
What do you think?

kriswest · 2026-03-25T12:19:43Z

To fix this properly we'd need to add these fields to the model and populate them, but maybe this is out of scope for this PR?

This might be a breaking change unless you include a migration of some sort for existing DBs or at least some form of backwards compatibility.

Maybe removing Date Modified and Date Created from the dropdown and open a separate issue to implement them correctly?

much simpler and non-breaking. If opening a ticket ensure it covers migration/consideration of how to make the change non-breaking (which would usually involve the ability to roll back!).

n.b. dates could possible be extracted from logs in existing records...

…arsePaginationParams

…pting search request

…enzi/git-proxy into server-side-pagination

fabiovincenzi · 2026-03-27T16:04:32Z

In this PR, list endpoints were changed from returning a plain array to returning { <entity>: [...], total: N }, and the data field was renamed to entity-specific keys to avoid the confusing response.data.data pattern. Both are technically breaking changes for any external client.

For the rename (data → pushes/repos/users), we could keep data as an alias. For the array, the only clean backwards-compatible solution would be to keep the existing endpoints returning the plain array and add new paginated endpoints alongside them, but that means maintaining duplicate logic until the old ones are deprecated.

Any thoughts?

kriswest · 2026-03-27T16:53:52Z

I think you've hit on the right answer there - you need to create new endpoints with the new behaviour then switch UI services and the UI itself over to them. Clearly mark the old ones are deprecated and consider where we can provide messaging about them being deprecated.

If we'd got this into the v2 scope, we wouldn't need to do that I guess... WDYT @jescalada

fabiovincenzi added 17 commits March 20, 2026 12:13

feat(db): add PaginationOptions and PaginatedResult types

7d5c31a

feat(db): add buildSearchFilter, buildSort and paginatedFind helpers

8541fb5

feat(db/file): implement paginated getRepos, getUsers, getPushes

2bb4bfa

feat(db/mongo): implement paginated getRepos, getUsers, getPushes

74895fd

refactor(db): update internal callers to destructure paginated results

bf9863c

feat(api): add parsePaginationParams and toPublicUser route utilities

2188282

feat(api): add pagination support to push route

a0a8b5d

feat(api): add pagination support to repo route

af7d290

feat(api): add pagination support to users route

16fac86

refactor(proxy): update proxy and checkUserPushPermission for paginat…

c76e4a1

…ed results

feat(ui): update git-push, repo, user services for paged responses

85e3666

feat(ui): add Pagination component

c5096ba

refactor(ui): migrate Filtering component to Material UI Select

d079512

feat(ui): wire pagination into PushesTable, UserList and Repositories…

7ab5852

… views

fix(cli): update getGitPushes to handle paginated API response

3a3891f

test: update existing tests for paginated responses

56922f9

test: add unit tests for buildSearchFilter, buildSort and parsePagina…

7716b4d

…tionParams

fabiovincenzi requested a review from a team as a code owner March 20, 2026 11:16

github-actions bot added the feature label Mar 20, 2026

test: update mongo integration tests for paginated responses

965efa0

fabiovincenzi added 3 commits March 20, 2026 12:33

test: add unit tests for paginatedFind in mongo helper

4c95924

test: update e2e push test for paginated getRepos response

e39c460

test: update Cypress commands for paginated repo API response

ab824d6

dcoric suggested changes Mar 22, 2026

View reviewed changes

kriswest linked an issue Mar 23, 2026 that may be closed by this pull request

Sort entries in the admin UI and apply proper pagination #1420

Open

kriswest mentioned this pull request Mar 23, 2026

23 March 2026 - GitProxy Meeting Minutes #1454

Open

30 tasks

roskosm1 reviewed Mar 23, 2026

View reviewed changes

re-vlad suggested changes Mar 23, 2026

View reviewed changes

src/db/mongo/helper.ts Outdated Show resolved Hide resolved

kriswest and others added 5 commits March 23, 2026 15:54

Merge branch 'main' into server-side-pagination

925c6bc

Merge branch 'main' into server-side-pagination

e7fc82f

fix: validate sortBy against per-endpoint allowlist to prevent sortin…

367226f

…g on sensitive fields

fix: guard against missing or non-string username body param in repo …

8b545ab

…patch routes

fix: load up to 100 users in AddUser dialog with integrated search in…

a68cb70

… select dropdown

jescalada requested changes Mar 24, 2026

View reviewed changes

fabiovincenzi self-assigned this Mar 24, 2026

fabiovincenzi added 3 commits March 24, 2026 19:54

fix: skip countDocuments for unpaginated internal queries

a611fb9

fix: debounce search input

c5ecc98

fix: cap skip at 10000 to prevent deep pagination queries

2c222a9

fabiovincenzi added 5 commits March 25, 2026 16:54

fix: rename paginated API response fields to entity-specific keys

a09117d

fix: keep Search mounted during loading to prevent pagination reset

eaaf5de

refactor: introduce PaginationQuery type to avoid repeated casts in p…

876b185

…arsePaginationParams

test: update tests to reflect renamed paginated response fields

259b51a

fix: remove Date Modified and Date Created from sort dropdown

6beb5f1

fabiovincenzi mentioned this pull request Mar 26, 2026

Add Date Modified and Date Created sorting for repositories #1486

Open

fabiovincenzi and others added 5 commits March 26, 2026 11:00

test: add cypress tests for search and pagination in repo list

212acdf

test: fix search cypress test to wait for initial load before interce…

846d33e

…pting search request

Merge branch 'main' into server-side-pagination

06b6af7

test: fix e2e test to use renamed repos field in getRepos response

8dc82dc

Merge branch 'server-side-pagination' of https://github.com/fabiovinc…

e737d6e

…enzi/git-proxy into server-side-pagination

fabiovincenzi requested review from dcoric, jescalada, re-vlad and roskosm1 March 27, 2026 15:53

Conversation

fabiovincenzi commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for endearing-brigadeiros-63f9d0 canceled.

Uh oh!

codecov bot commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dcoric left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

roskosm1 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jescalada left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fabiovincenzi commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kriswest commented Mar 25, 2026

Uh oh!

fabiovincenzi commented Mar 27, 2026

Uh oh!

kriswest commented Mar 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

fabiovincenzi commented Mar 20, 2026 •

edited

Loading

netlify bot commented Mar 20, 2026 •

edited

Loading

codecov bot commented Mar 20, 2026 •

edited

Loading

roskosm1 left a comment •

edited

Loading

jescalada left a comment •

edited

Loading

fabiovincenzi commented Mar 25, 2026 •

edited

Loading