#435: checkpoint properties based querying by lsulak · Pull Request #457 · AbsaOSS/atum-service

lsulak · 2026-07-21T11:26:40Z

Overview

What's changed:

Added new versions of the flows.get_flow_checkpoints and runs.get_partitioning_checkpoints functions, each now supporting an optional i_checkpoint_properties HSTORE parameter to filter checkpoints by matching property key-value pairs. This enables more flexible and precise queries for checkpoints.
Updated function signatures and dropped previous overloads to avoid ambiguity with named/default arguments.

Release Notes

Adding support for filtering checkpoints by arbitrary checkpoint properties, improving the querying capabilities for both flows and partitionings. This is usually part of GET endpoints that use base64 encoded key-value map containing the properties, but it's also available via Atum Reader.
One real example usage for this is that our spark pipelines generate a unique ExecutionID and that is stored there. We want to query checkpoints based on the ExecutionID.
Added i_latest_first parameter support to runs.get_partitioning_checkpoints, propagating all the way from DB function to REST API layer, to manipulate with the order of checkpoints being retrieved.

File Path	Coverage	Threshold	Status
CheckpointController.scala	88.89%	60.0%	✅
CheckpointControllerImpl.scala	86.46%	60.0%	✅
CheckpointRepository.scala	100.0%	60.0%	✅
CheckpointRepositoryImpl.scala	97.25%	60.0%	✅
CheckpointService.scala	100.0%	60.0%	✅
CheckpointServiceImpl.scala	94.12%	60.0%	✅
Endpoints.scala	98.8%	60.0%	✅
FlowController.scala	100.0%	60.0%	✅
FlowControllerImpl.scala	73.33%	60.0%	✅
FlowReader.scala	92.89%	60.0%	✅
FlowRepository.scala	100.0%	60.0%	✅
FlowRepositoryImpl.scala	86.89%	60.0%	✅
FlowService.scala	100.0%	60.0%	✅
FlowServiceImpl.scala	90.32%	60.0%	✅
GetFlowCheckpoints.scala	87.5%	60.0%	✅
GetPartitioningCheckpoints.scala	86.4%	60.0%	✅
JsonSyntaxExtensions.scala	86.96%	60.0%	✅
PartitioningReader.scala	92.86%	60.0%	✅

Run 29999546289 · Event: pull_request

…ith how we treat partitioning for example), and fixing decoder to URL decoder

Copilot

Pull request overview

Adds end-to-end support for filtering flow/partitioning checkpoint listings by arbitrary checkpoint property key/value pairs, propagating the new optional filter through HTTP → controller/service/repository → DB functions, and updating tests accordingly.

Changes:

Add checkpointProperties filter support to v2 endpoints (checkpoint-properties query param as base64url-encoded JSON) and propagate it through server layers.
Extend DB functions flows.get_flow_checkpoints and runs.get_partitioning_checkpoints to accept an optional HSTORE filter and apply AND-semantics matching.
Update/extend server, reader, model, and database integration tests to cover the new filter and encoding/decoding behavior.

Reviewed changes

Copilot reviewed 38 out of 38 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
server/src/test/scala/za/co/absa/atum/server/api/v2/service/FlowServiceUnitTests.scala	Updates mocks/calls for new `checkpointProperties` argument.
server/src/test/scala/za/co/absa/atum/server/api/v2/service/CheckpointServiceUnitTests.scala	Updates mocks/calls for new `checkpointProperties` argument.
server/src/test/scala/za/co/absa/atum/server/api/v2/repository/FlowRepositoryUnitTests.scala	Updates function-arg expectations for new DB function signature.
server/src/test/scala/za/co/absa/atum/server/api/v2/repository/CheckpointRepositoryUnitTests.scala	Updates function-arg expectations for new DB function signature.
server/src/test/scala/za/co/absa/atum/server/api/v2/http/GetPartitioningCheckpointsEndpointUnitTests.scala	Adapts endpoint logic/mocks for `checkpoint-properties` query param.
server/src/test/scala/za/co/absa/atum/server/api/v2/http/GetFlowCheckpointsEndpointUnitTests.scala	Adds endpoint tests for properties filtering + invalid base64 handling.
server/src/test/scala/za/co/absa/atum/server/api/v2/controller/FlowControllerUnitTests.scala	Updates controller calls for new `checkpointProperties` argument.
server/src/test/scala/za/co/absa/atum/server/api/v2/controller/CheckpointControllerUnitTests.scala	Updates controller calls for new `checkpointProperties` argument.
server/src/test/scala/za/co/absa/atum/server/api/database/runs/functions/GetPartitioningCheckpointsIntegrationTests.scala	Updates integration args to include `checkpointProperties`.
server/src/test/scala/za/co/absa/atum/server/api/database/flows/functions/GetFlowCheckpointsIntegrationTests.scala	Updates integration args to include `checkpointProperties`.
server/src/main/scala/za/co/absa/atum/server/api/v2/service/FlowServiceImpl.scala	Propagates `checkpointProperties` to repository call.
server/src/main/scala/za/co/absa/atum/server/api/v2/service/FlowService.scala	Extends service API with `checkpointProperties`.
server/src/main/scala/za/co/absa/atum/server/api/v2/service/CheckpointServiceImpl.scala	Propagates `checkpointProperties` to repository call.
server/src/main/scala/za/co/absa/atum/server/api/v2/service/CheckpointService.scala	Extends service API with `checkpointProperties`.
server/src/main/scala/za/co/absa/atum/server/api/v2/repository/FlowRepositoryImpl.scala	Passes `checkpointProperties` into DB function args.
server/src/main/scala/za/co/absa/atum/server/api/v2/repository/FlowRepository.scala	Extends repository API with `checkpointProperties`.
server/src/main/scala/za/co/absa/atum/server/api/v2/repository/CheckpointRepositoryImpl.scala	Passes `checkpointProperties` into DB function args.
server/src/main/scala/za/co/absa/atum/server/api/v2/repository/CheckpointRepository.scala	Extends repository API with `checkpointProperties`.
server/src/main/scala/za/co/absa/atum/server/api/v2/http/Endpoints.scala	Adds `checkpoint-properties` query param + base64url JSON codec for Map filter.
server/src/main/scala/za/co/absa/atum/server/api/v2/controller/FlowControllerImpl.scala	Propagates `checkpointProperties` to service call.
server/src/main/scala/za/co/absa/atum/server/api/v2/controller/FlowController.scala	Extends controller API with `checkpointProperties`.
server/src/main/scala/za/co/absa/atum/server/api/v2/controller/CheckpointControllerImpl.scala	Propagates `checkpointProperties` to service call.
server/src/main/scala/za/co/absa/atum/server/api/v2/controller/CheckpointController.scala	Extends controller API with `checkpointProperties`.
server/src/main/scala/za/co/absa/atum/server/api/database/runs/functions/GetPartitioningCheckpoints.scala	Extends DB function wrapper args with `checkpointProperties`.
server/src/main/scala/za/co/absa/atum/server/api/database/flows/functions/GetFlowCheckpoints.scala	Extends DB function wrapper args with `checkpointProperties`.
reader/src/test/scala/za/co/absa/atum/reader/PartitioningReaderUnitTests.scala	Adds reader test verifying properties filter query param is sent.
reader/src/test/scala/za/co/absa/atum/reader/FlowReaderUnitTests.scala	Adds reader test verifying properties filter query param is sent.
reader/src/main/scala/za/co/absa/atum/reader/requests/QueryParamNames.scala	Adds `checkpoint-properties` query parameter name constant.
reader/src/main/scala/za/co/absa/atum/reader/PartitioningReader.scala	Adds `getCheckpointsByPropertiesPage` and sends encoded filter param.
reader/src/main/scala/za/co/absa/atum/reader/FlowReader.scala	Adds `getCheckpointsByPropertiesPage` and sends encoded filter param.
model/src/test/scala/za/co/absa/atum/model/utils/JsonSerializationSyntaxUnitTests.scala	Adds test for base64url JSON encoding of properties Map.
model/src/test/scala/za/co/absa/atum/model/utils/JsonDeserializationSyntaxUnitTests.scala	Adds tests for base64url decoding + round-trip behavior.
model/src/main/scala/za/co/absa/atum/model/utils/JsonSyntaxExtensions.scala	Switches base64 decoding to URL-safe decoder.
jmf-rules.txt	Updates rules list to include `QueryParamNames`.
database/src/test/scala/za/co/absa/atum/database/runs/GetPartitioningCheckpointsIntegrationTests.scala	Adds integration test validating DB-side properties filtering.
database/src/test/scala/za/co/absa/atum/database/flows/GetFlowCheckpointsIntegrationTests.scala	Adds integration test validating DB-side properties filtering (incl. AND semantics).
database/src/main/postgres/runs/V0.7.0.2__get_partitioning_checkpoints.sql	New migration adding HSTORE properties filter to partitioning checkpoints function.
database/src/main/postgres/flows/V0.7.0.1__get_flow_checkpoints.sql	New migration adding HSTORE properties filter to flow checkpoints function.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

… SQL functions - Add ORDER BY process_start_time DESC, id_checkpoint ASC to _has_more subquery for deterministic pagination results in both functions - Fix limited_checkpoints CTE: correct ORDER BY from id_checkpoint-first to process_start_time DESC, id_checkpoint ASC in get_partitioning_checkpoints - Add id_checkpoint ASC tie-breaker to limited_checkpoints and final ORDER BY in get_flow_checkpoints for stable pagination when timestamps collide

Copilot

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated 2 comments.

lsulak · 2026-07-23T10:54:43Z

Review from Copilot (twice) finished, deployed and tested via Postman, and also ran integration & agent-server compatibility tests, it all passes: https://github.com/absa-group/atum-service-deployment/actions/runs/29994664014

#435: checkpoint properties based querying

6031582

lsulak self-assigned this Jul 21, 2026

lsulak requested review from ABLL526, MatloaItumeleng, Zejnilovic, benedeki, dk1844 and salamonpavel as code owners July 21, 2026 11:26

lsulak added 4 commits July 21, 2026 16:15

coverage ignore update

f6b2ae5

coverage ignore update & code doc

41fb6ff

coverage ignore update & code doc

4569e96

base64 instead of url encoded raw checkpoint properties (consistent w…

a5d3bfd

…ith how we treat partitioning for example), and fixing decoder to URL decoder

lsulak requested a review from Copilot July 22, 2026 13:31

Copilot started reviewing on behalf of lsulak July 22, 2026 13:31 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Copilot started work on behalf of lsulak July 22, 2026 15:16 View session

Copilot finished work on behalf of lsulak July 22, 2026 15:20

Copilot started work on behalf of lsulak July 22, 2026 15:21 View session

Copilot finished work on behalf of lsulak July 22, 2026 15:21

Copilot started work on behalf of lsulak July 22, 2026 15:30 View session

Copilot finished work on behalf of lsulak July 22, 2026 15:31

Copilot started work on behalf of lsulak July 22, 2026 15:32 View session

Copilot finished work on behalf of lsulak July 22, 2026 15:33

lsulak added 3 commits July 22, 2026 17:36

consistency in checkpoint retrieval

9466a43

consistency in checkpoint retrieval

d376135

post-refactoring cleanup

42a0469

lsulak requested a review from Copilot July 22, 2026 15:51

post-refactoring cleanup

6932347

Copilot started reviewing on behalf of lsulak July 22, 2026 15:52 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

fixing unit tests

0586721

lsulak requested a review from Copilot July 23, 2026 07:57

Copilot started reviewing on behalf of lsulak July 23, 2026 07:58 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

lsulak added 2 commits July 23, 2026 10:17

fixing integration tests

22b41bb

redundant

5074d2a

lsulak requested a review from Copilot July 23, 2026 08:50

Copilot started reviewing on behalf of lsulak July 23, 2026 08:51 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread ...t/scala/za/co/absa/atum/server/api/v2/http/GetPartitioningCheckpointsEndpointUnitTests.scala

Comment thread database/src/main/postgres/runs/V0.7.0.2__get_partitioning_checkpoints.sql

fixing tests & forgotten param propagation to outer layers

020d945

lsulak merged commit 31c64a4 into master Jul 23, 2026
9 checks passed

lsulak deleted the feature/435-checkpoint-filtering-by-its-properties branch July 23, 2026 10:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

#435: checkpoint properties based querying#457

#435: checkpoint properties based querying#457
lsulak merged 14 commits into
masterfrom
feature/435-checkpoint-filtering-by-its-properties

lsulak commented Jul 21, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 21, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

lsulak commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

lsulak commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Release Notes

Related

Uh oh!

github-actions Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

lsulak commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lsulak commented Jul 21, 2026 •

edited

Loading

github-actions Bot commented Jul 21, 2026 •

edited

Loading