feat: make imports validated, auditable, and resumable - #1298
Merged
Baskarayelu merged 1 commit intoAug 30, 2026
Merged
Baskarayelu merged 1 commit into
Baskarayelu merged 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
accepted,retryable, andrejectedoutcomes without echoing sensitive row values.207for partial persistence and expose operation, accepted/rejected/retried counts.import_rows_total{status}metrics without payload data in labels.Acceptance criteria
422with zero committer calls; commit routes requireADMIN_WRITEand tenant context.207.Security and correctness note
Idempotency keys are scoped by trusted tenant context and bound to the complete file plus mapping fingerprint. Reusing a key for different content returns
409 IdempotencyConflict. The implementation never includes addresses, emails, names, credentials, or adapter exception text in row outcomes or metric labels.A full validation pass occurs before checkpoint creation or persistence. During partial persistence, an adapter failure marks only that row retryable; it cannot cause already accepted rows to be submitted again. Per-key locking prevents concurrent same-key calls from racing before an accepted outcome is recorded.
The current repository uses an in-memory checkpoint store. The documentation specifies the shared database/coordination adapter and provider reconciliation needed for cross-worker and crash-safe production deployment.
Compatibility and rollback
Existing
commitImportFilecallers remain valid because new options are optional. Existing dry-run and validation response fields remain intact, with additive row-outcome data. The commit response now includes operation metadata and returns207when persistence is partial. No migration is required for the current in-memory store; a future persistent checkpoint table should be additive and deployed before writers. Rollback is schema-safe but reverts operation-aware responses and resume protection.Validation
npx vitest run src/services/imports/commit.test.ts src/services/imports/resumableCommit.test.ts— 13 passing, 0 failingnpx tsc --noEmitfiltered to changed import/route/schema files — no errorsgit diff --check— passedsrc/db/repositories/auditChainVerificationRepository.tsandsrc/db/repositories/baseRepository.tsRelated issue
Closes #1241