Cloud script and test

[marlens123]

Description: This PR adds functions for extracting cloud-related metadata from a given evaluation .tif tile. The extracted values will allow us to group performance based on the presence of clouds.

Contents:

• Bitwise extraction of cloud, cloud shadow, and cirrus flags from MODIS QA state integers.
• Methods to compute daily and cumulative cloud metrics (e.g., total cloudy days, last clear day) using majority voting across multiple observations per day.

Additional information:
The evaluation .tif file will contain all satellite data + auxiliary variables from the preceding week, which we use as predictors for the AI model. The shape is [all_combined_bands, H, W], where all_combined_bands includes all dynamic-in-time bands interleaved for all timesteps followed by the static-in-time bands.
dynamic-in-time bands means acquisitions that change over the week (e.g., Landsat, MODIS, Sentinel-1, -2, -3), static-in-time bands means acquisitions that stay constant over the week (e.g., topography, landcover).
Sample tif files are contained in data/eval_tifs.

[djmallum]

I'm not convinced that bitwise_extract works as expected. Consider writing some tests for it. Likewise, include a wider range of outcomes for your test of map_int_to_cloud_states. You could look for ones that return all possible expected states and a few ways to challenge each. The expected states are a 3 bit binary integer, from 0 (000) to 7 (111). So you should have tests for each.

Use my suggest method for converting strings in binary form to integers to manufacture integers to test each. That way the tests are more clear.

[djmallum]

I noticed here that your tests are fairly similar. Targeting those with mixed cloudy conditions each time.

I think if you are still expecting odd results, I would consider doing two things:

1. Testing your bitwise_extract function separately.
2. Supplying a few more unique possible states.

[djmallum]

Bitwise operations certainly are elegant, but unless speed is a large factor, I might consider something more intuitive and easier to debug. I spent some time looking at this and how bitwise operations work in python and I'm still not really sure why this function works.

As I suggested elsewhere, I would design tests for this function. Choose some convenient binary representations. A nice way to do this is to write your tests as a string of binary representations, such as:

test_value1 = int('1101011101',2)

And you can more easily look at the binary representation for the tests.

I tried walking through the steps of this function, and I'm not convinced it works. That could be because of a misunderstanding with the method. But for example, with value = 1020, this has a binary representation of 1111111100. And if my understanding is correct, then this should return a cirrus of none because digits 8 and 9 are both 0. But when I run the value through the code with from_bit=8 and to_bit=9, I get a value of 4 and this returns true.

Follow my advice in a later comment, could it be that this example is simply missed because the range of tests are too similar?

[marlens123]

Thank you for the helpful comments!
I’ve now added more tests for different cloud state combinations, and checks for the binary representations themselves. All tests pass. I also found more blogposts that use the same extract_bitwise function which makes me more confident that the function works as intended. The tricky part here is the bit ordering: The MODIS user guide specifies that the 1km_state variable is stored as a 16-bit unsigned integer, with bit 0 being the least significant bit (LSB), so the right-most bit. For the 1020 example this means a binary of '0000001111111100', hence bit 8 and 9 are both 1

[marlens123]

1020 for from_bit=8 and to_bit=9 returns 4 for you? I get a 3