To support extremely large simulations with dense station grids we should introduce chunking using dask. The fastest way to do this would be:
Use xarray's map_blocks api, which allows us map over chunks in serial. Unlike hf sim, we would want to process these in serial because there is already a parallel layer in the rust code that processes the intensity measures using rayon. Rayon has a much lower parallel overhead than dask so this is preferable to parallelising at the dask level.
To support extremely large simulations with dense station grids we should introduce chunking using dask. The fastest way to do this would be:
Use xarray's map_blocks api, which allows us map over chunks in serial. Unlike hf sim, we would want to process these in serial because there is already a parallel layer in the rust code that processes the intensity measures using rayon. Rayon has a much lower parallel overhead than dask so this is preferable to parallelising at the dask level.