Guidance on cd_values and cd_nelems

[hmaarrfk]

Is the goal of cd_values and cd_nelems to help decode or reencode with the exact same parameters?

I'm thinking of:

• load data
• transform data
• Store transformed data

and perhaps you want to store in the "exact same way as b4".

You are welcome! Remember to contact again once the cd_nelems and cd_values[] of your filter plugin are defined. This info is the essential part of a filter plugin definition.

Originally posted by @ajelenak in #215 (comment)

If it is just about "reading" data, i think perhaps there may not be much needed?????

Can these be variable lengthed???

[ajelenak]

The primary goal of cd_values[] and cd_nelems is to allow user application to configure the filter how to process chunk bytes when writing (storing) them in the file. They are also recorded in the dataset's creation property list for whatever they may be needed later.

By decode I assume you mean apply filter on the chunk bytes when reading, i.e. decompress the chunk? I think in the most general case, it depends on the filter. Typically there should be enough information in the filtered chunk bytes for the filter to be able to recreate the original bytes.

• Store transformed data

Where? In a new dataset? You could pull out the cd_values[] from another dataset and use them to store these transformed data with the same filter configuration in a new dataset.

Can these be variable lengthed???

Don't understand this.

[hmaarrfk]

ok, so if i understand correctly, may compression algorithms such as zlib and zstd include a 'level' parameter, which would be some integer number that describes how much computation to use to compress things.

cd_values would hold this parameter, so that the user application can recompress data with the same amount of effort.

Can these be variable lengthed???

sometimes there are multiple parameters, and sometimes the numbers of parameters "grows" as one learns of the parameters for the compression.

This is especially true with compression algorithms that try to take advantage of "2D" aspects of data.

For example, JPEG2000 (lossless) provides things like:

• Patch size
• Precinct size
  which would tune at least "2 parameters".

So in some cases:

• Specify 0 parameters
• Specify 1 parameter
• Specify 2 parameters
• Specify "3 parameters" when I learn about more parameters.

[ajelenak]

So in some cases:

• Specify 0 parameters
• Specify 1 parameter
• Specify 2 parameters
• Specify "3 parameters" when I learn about more parameters.

Yes, this is possible. The cd_nelems value tells how many parameters are given in cd_values[]. It's up to the filter plugin how to treat the cd_values[] parameters accordingly. Each positional parameter of cd_values[] can always be the same, or different based on their total number. But all of this logic must be documented for data interoperability purposes.

[hmaarrfk]

got it thanks. is there a good example of a 'complex' filter taht I can model these cd_values and cd_nelems after?

[ajelenak]

Perhaps the BLOSC plugin. The cd_values[] are really based on the filter, what parameters are required or optional, and how best to expose them to users.