Currently sink reducers will list recursively. For large assets dirs this means listing can take a long time. on the other hand, filename templates for parquet/jsonl can also be nested inside some folder... We need a more efficient reducer listing setup (rmtrees for assets folders already work well enough)
Currently sink reducers will list recursively. For large assets dirs this means listing can take a long time. on the other hand, filename templates for parquet/jsonl can also be nested inside some folder... We need a more efficient reducer listing setup (rmtrees for assets folders already work well enough)