Problem
The writer appears to count rows_written incorrectly. The metric should be incremented only after a row has actually been written/persisted by the writer.
Current observed behavior from step metrics suggests rows_written is not tracking real successful writes: the metric stays near zero for most of the run and then spikes, while files_written remains steady around 12-14 files. This makes writer throughput and progress telemetry misleading.
Expected behavior
rows_written increments only when the writer has successfully written a row.- Rows that are queued, batched, attempted, skipped, failed, or rolled back must not be counted as written.
- If writing happens in batches/files, the metric should be emitted after the successful write/flush/commit point for the rows represented by that batch.
Notes
Screenshots from the worker step metrics show files_written behaving continuously while rows_written reports a single large spike, which suggests row accounting is being emitted at the wrong stage or with the wrong aggregation semantics.
Problem
The writer appears to count
rows_writtenincorrectly. The metric should be incremented only after a row has actually been written/persisted by the writer.Current observed behavior from step metrics suggests
rows_writtenis not tracking real successful writes: the metric stays near zero for most of the run and then spikes, whilefiles_writtenremains steady around 12-14 files. This makes writer throughput and progress telemetry misleading.Expected behavior
rows_writtenincrements only when the writer has successfully written a row.Notes
Screenshots from the worker step metrics show
files_writtenbehaving continuously whilerows_writtenreports a single large spike, which suggests row accounting is being emitted at the wrong stage or with the wrong aggregation semantics.