Skip to content

feat: make imports validated, auditable, and resumable - #1298

Merged
Baskarayelu merged 1 commit into
CredenceOrg:mainfrom
greatest0fallt1me:fix/1241-resumable-import-validation
Aug 30, 2026
Merged

Baskarayelu merged 1 commit into
CredenceOrg:mainfrom
greatest0fallt1me:fix/1241-resumable-import-validation

Conversation

@greatest0fallt1me

Copy link
Copy Markdown
Contributor

Summary

  • Keep the existing full-file dry run as a zero-write validation gate before any committer call.
  • Add tenant-scoped idempotency keys, SHA-256 fingerprints, stable row keys, and per-key serialization.
  • Return row-level accepted, retryable, and rejected outcomes without echoing sensitive row values.
  • Resume only retryable rows; accepted rows are replayed from the checkpoint and never upserted again.
  • Return 207 for partial persistence and expose operation, accepted/rejected/retried counts.
  • Add import_rows_total{status} metrics without payload data in labels.
  • Require trusted tenant context on commit routes and document validation, failure, migration, rollback, and recovery behavior.

Acceptance criteria

  • Schema, authorization, tenant, and business-rule validation happen before irreversible writes. Invalid commits return 422 with zero committer calls; commit routes require ADMIN_WRITE and tenant context.
  • The validation/persistence boundary is documented: validation is atomic, persistence is explicitly partial per row, and partial failures return 207.
  • Rows have stable fingerprint-derived idempotency keys and deterministic resume behavior. Same-key concurrent requests are serialized.
  • Operators receive row numbers, stable row keys, safe machine-readable codes, operation IDs, and accepted/rejected/retried counts without sensitive values.

Security and correctness note

Idempotency keys are scoped by trusted tenant context and bound to the complete file plus mapping fingerprint. Reusing a key for different content returns 409 IdempotencyConflict. The implementation never includes addresses, emails, names, credentials, or adapter exception text in row outcomes or metric labels.

A full validation pass occurs before checkpoint creation or persistence. During partial persistence, an adapter failure marks only that row retryable; it cannot cause already accepted rows to be submitted again. Per-key locking prevents concurrent same-key calls from racing before an accepted outcome is recorded.

The current repository uses an in-memory checkpoint store. The documentation specifies the shared database/coordination adapter and provider reconciliation needed for cross-worker and crash-safe production deployment.

Compatibility and rollback

Existing commitImportFile callers remain valid because new options are optional. Existing dry-run and validation response fields remain intact, with additive row-outcome data. The commit response now includes operation metadata and returns 207 when persistence is partial. No migration is required for the current in-memory store; a future persistent checkpoint table should be additive and deployed before writers. Rollback is schema-safe but reverts operation-aware responses and resume protection.

Validation

  • npx vitest run src/services/imports/commit.test.ts src/services/imports/resumableCommit.test.ts — 13 passing, 0 failing
  • npx tsc --noEmit filtered to changed import/route/schema files — no errors
  • git diff --check — passed
  • 679 lines added across implementation, tests, metrics, schemas, and documentation
  • Full build remains blocked by pre-existing invalid-character errors in unrelated src/db/repositories/auditChainVerificationRepository.ts and src/db/repositories/baseRepository.ts

Related issue

Closes #1241

@Baskarayelu
Baskarayelu merged commit a67736b into CredenceOrg:main Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Quality][Medium] Make import validation transactional and resumable

2 participants