Use of Brunslicompression by default in mrf-jpeg-compressed data

[gauteaasolaas]

One of my colleagues tried to use standard GDAL to read our recent mrf-encoded orthofiles and found that it couldn't.
After some investigations he found that the mrf-files have Brunsli-compression and had to make a new build of gdal where this was included.

A couple of questions arise:

• If Brunslicompression is the correct thing to do, why is it not included in GDAL by default?
• If it is not, then why is it used by default by OptimizeRasters?
• How can I force OptimizeRasters to skip the Brunslicompression . A setting in a config-file?

https://github.com/Esri/OptimizeRasters/blob/master/GDAL/bin/brunslidec-c.dll

ChatGPT gave me the summary blow. It confirms that this is limiting the type of clients that can read the files:

• Not suitable where you need a format natively supported by browsers/clients — it’s an extra step (encode/decode) and requires Brunsli support on both ends.
• Adoption is limited — ecosystem/tools are smaller than for JPEG/PNG/WebP/etc.

ChatGPT:
Do you mean the Brunsli JPEG-recompression tool from Google Research? I can explain what it is, how it works at a high level, how to use it, and tradeoffs — or help compress some images if you want. Summary first, then options.
What Brunsli is
Brunsli is a lossless JPEG-recompression format/tool (developed by Google Research) that re-encodes an existing JPEG bitstream into a more compact representation. After decompression you get back the original JPEG bitstream (and therefore the identical decoded image).
It is NOT a new image format for browsers; it’s a way to store/transmit existing JPEGs more efficiently while preserving exact JPEG decoding semantics.
How it works (high level)
It parses the JPEG file (DCT coefficients, quantization tables, metadata, Huffman tables, etc.), removes redundancy in how coefficients and tables are represented, and re-encodes the data using more efficient models and entropy coding than baseline JPEG’s canonical Huffman representation.
The result is a Brunsli blob that can be losslessly converted back to the original JPEG bytes (so any standard JPEG decoder yields the same pixels).
The algorithm uses context modeling tailored to JPEG coefficient statistics (not simply generic gzip/deflate).
Why/when to use it
Good for archival or transmission where you must keep an exact JPEG (or exact pixel output) but want to reduce bytes.
Typical savings depend on image content and existing JPEG encoder; many images see meaningful savings vs. the original JPEG and versus standard compressors (gzip/zlib), although exact numbers vary.
Not suitable where you need a format natively supported by browsers/clients — it’s an extra step (encode/decode) and requires Brunsli support on both ends.
Tradeoffs
Compression is more CPU/time intensive than running a standard lossless compressor; decompression is generally faster but still more work than a simple unzip.
Adoption is limited — ecosystem/tools are smaller than for JPEG/PNG/WebP/etc.
Good when you control both ends (server+client) or when storing images for later retrieval.
Using Brunsli (CLI)
The official source and binaries are on the Google brunsli repo: https://github.com/google/brunsli
Typical usage (examples — check your installed version’s --help/README to confirm exact flags):
Encode (JPEG -> brunsli): brunsli -encode input.jpg output.brunsli
Decode (brunsli -> JPEG): brunsli -decode input.brunsli output.jpg
Most builds include a binary and a library. On Linux/macOS you can build from source with the repo’s instructions (cmake or bazel depending on the version).
Programmatic use
The repo exposes a C++ library you can link into your programs to compress/decompress JPEGs.
There are third-party bindings for some languages (search for “brunsli” + your language). If you need a script, I can provide an example (bash, Python wrapper calling the CLI, or C++ snippet).
Alternatives and comparisons
Compared to general-purpose compressors (gzip, zstd on raw JPEG) Brunsli usually wins because it understands JPEG internals.
It’s conceptually different from re-encoding the image using a different lossy encoder (e.g., WebP/AVIF/MozJPEG) — those change the image pixels or quantization and may produce smaller files but are lossy or require consumers to support the new format.
For aggressive byte-size reduction with acceptable visual changes, re-encoding to modern image formats (AVIF/WebP) is often better for web delivery; use Brunsli when the JPEG must be preserved.

[gauteaasolaas]

Additional question:

• If we need to get rid of the Brunslicompression, do we have a tool that can do this for us?

[lucianpls]

@gauteaasolaas,

ChatGPT is not quite right, it went off-track, every time it mentions zip, zlib, those are not applicable to this subject.

Brunsli is indeed a standard JPEG in a smaller footprint, saving 22% of space vs JFJF. For large data collections on the cloud, that 22% less storage and bandwidth is huge, especially because it's still the same JPEG and not a new format with different characteristics. It is also fast, in most cases the use of brunsli can even be faster that plain JFIF due to reading less data.
That is the reason why brunsli is the default JPEG packing in MRF. It makes sense from the point of view of OptimizeRasters, a 22% smaller JPEG is a very valuable "optimization". Not using it means paying 28% more than necessary for storage and bandwidth.

As far as the inclusion in GDAL, it is part of the source and it's not that hard to include in the build. If I recall correctly, it's not used by default because at the time it was added, 5 years ago, the brunsli bitstream was supposed to be one of the JPEG - XL profiles. That is no longer the case, they are no longer binary compatible, so maybe it is time to re-enable brunsli in gdal by default.

If you really want to accept the 28% extra cost, there is a MRF create option setting used to generate the JFIF JPEG format, "-co OPTIONS=JFIF:1", I'm pretty sure that can be included in a OR template, under GDAL_Translate_UserParameters.
As far as conversion of existing MRFs, The "jxl" binary built from https://github.com/nasa-gibs/mrf/blob/master/mrf_apps/jxl.cpp does a lossless MRF conversion JFIF<->brunsli.
You can also use gdal_translate (or arcpy w/gdal) with the above create option to either re-create the MRF or convert the brunsli MRF version into a JFIF MRF. That will be a very slightly lossy conversion, not noticeable, since there is no way to even read JPEG pixels without introducing lossy artifacts.

[gauteaasolaas]

For users like me it would be very much appreciated if you could initiate the inclusion of the Brunsli-compression in the standard GDAL package. I know that the guys here are working on using our imagery for AI-model training using Esri tools, so we would need it to be activated within the Conda-environment used by these. (if I understood it right).
Given the benefits you explained I guess we will keep using it in our MRF-conversion. 22% adds up both in storage and transfer.

[genericuser3462]

Hi there, from testing the default MRF output with JPEG compression (using OptimizeRasters 2.3.0) the imagery isn't visible in Esri ArcGIS Pro 3.4 + or Server 11.4 +. It works fine in Pro 3.1-3.3 and Server 11.1. In comparison MRFs compressed using JFIF do render correctly.

I appreciate this question is perhaps more related to paid-for Esri software than OptimizeRasters, but presumably there's a reasonable overlap in people using OptimizeRasters to view imagery in ArcGIS software?

[abhiataero]

@genericuser3462 this is a known issue in ArcGIS Pro, we are working towards to get patch for this