Background
The current parallel mode can corrupt data if a Delta Lake format is used without a transaction coordinator.
This can be fixed by ensuring the same table is never written in parallel.
Feature
Add a parallelism mode that allows parallel job runs, but never the same job for different info dates.
Example [Optional]
--
Proposed Solution [Optional]
--
Background
The current parallel mode can corrupt data if a Delta Lake format is used without a transaction coordinator.
This can be fixed by ensuring the same table is never written in parallel.
Feature
Add a parallelism mode that allows parallel job runs, but never the same job for different info dates.
Example [Optional]
--
Proposed Solution [Optional]
--