HTTP server that exposes stores, arrays, groups

[d-v-b]

This PR adds an experimental http server in experimental.serve. This server can expose stores over http. It can also expose arrays and groups over http. Exposing a store means exposing the entire key: value space of the store. Exposing an array means only exposing the metadata + chunks. Exposing a group means only exposing sub-groups and sub-arrays. See #3731 for more on this distinction.

The server is an optional dependency implemented via starlette. It handles byte-range reads and other http methods. CORS headers and allowed methods can be configured. I'm considering handling prefix requests like foo/bar by returning a simple HTML document that lists the visible keys under foo/bar/, for user-friendliness and to aid httpstore readers that use such responses for listing contents. I'd also like to implement convenience functions for kicking off a server from jupyter and a CLI.

Opening as a draft while I work on this.

[d-v-b]

changes in this file are simplifications to our examples testing infrastructure. Instead of re-writing the script header, we just override the declared zarr dep in the invocation of uv run ...

[d-v-b]

we eventually need to find a more natural place for this code. I'm not sure which module it should live in.

[d-v-b]

I'm considering handling prefix requests like foo/bar by returning a simple HTML document that lists the visible keys under foo/bar/, for user-friendliness and to aid httpstore readers that use such responses for listing contents. I'd also like to implement convenience functions for kicking off a server from jupyter and a CLI.

Didn't end up doing these things. We can add them later if people are interested.

[d-v-b]

here's a demo of this functionality:

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "zarr[server] @ git+https://github.com/d-v-b/zarr-python@feat/experimental-server",
# ]
# ///

import zarr
import numpy as np
from zarr.storage import MemoryStore
from zarr.experimental.serve import serve_node

data = np.arange(1000, dtype='uint8').reshape(10, 10, 10)
store = MemoryStore({})
z = zarr.create_array(store, data=data, write_data=True)
serve_node(
    z,
    host="127.0.0.1",
    port=8000,
    cors_options={"allow_origins": ["*"], "allow_methods":["GET", "HEAD"]}
    )

If you run that script, then visit https://neuroglancer-demo.appspot.com/#!%7B%22dimensions%22:%7B%22dim_0%22:%5B1%2C%22%22%5D%2C%22dim_1%22:%5B1%2C%22%22%5D%2C%22dim_2%22:%5B1%2C%22%22%5D%7D%2C%22position%22:%5B1.5%2C4.5%2C5.5%5D%2C%22crossSectionScale%22:0.03421811831166599%2C%22projectionOrientation%22:%5B-0.06506630778312683%2C-0.14447830617427826%2C-0.003086749231442809%2C0.9873615503311157%5D%2C%22projectionScale%22:53.328548429788874%2C%22layers%22:%5B%7B%22type%22:%22image%22%2C%22source%22:%22http://127.0.0.1:8000/%7Czarr3:%22%2C%22tab%22:%22source%22%2C%22name%22:%228000%22%7D%5D%2C%22selectedLayer%22:%7B%22visible%22:true%2C%22layer%22:%228000%22%7D%2C%22layout%22:%224panel-alt%22%7D, you should see the array data

[dcherian]

Does this really need to belong in the core python library? Is there any advantage to experimenting with it in this repository?

[d-v-b]

Does this really need to belong in the core python library? Is there any advantage to experimenting with it in this repository?

IMO yes.

• if we ever need to test an HTTP store, we would need a way to expose a zarr store over HTTP, e.g. the stuff in this PR.
• many zarr datasets are too big to visualize with tools like matplotlib. This tool makes it easy to expose Zarr data to web-based visualization tools like neuroglancer, fulfilling a basic need for zarr users.

[d-v-b]

and putting this in experimental is low commitment. if nobody uses this feature and it's a development burden, we can remove it. If on the other hand it's actually useful, we can keep it.

[d-v-b]

another important use case: in zarr-python today, if you create a custom store, there is currently no way in zarr python to expose that store as a writable endpoint to a zarr-aware client. This PR enables this functionality.

[psobolewskiPhD]

Love this, as someone who regularly uses @manzt https://github.com/manzt/simple-zarr-server
@kephale also has an implementation in a script!
Super handy to be able to do visualization of remote data, particularly in headless environments.

[d-v-b]

thanks @psobolewskiPhD, given your experience with other tools let me know if there are any features missing from this PR and I can add them.

[dcherian]

many zarr datasets are too big to visualize with tools like matplotlib.

I don't think this isn't a good argument. many zarr datasets are too big to compute on; should we vendor cubed/dask too?

That said, i don't plan on helping maintain it so ... no skin off my back hehe

[d-v-b]

I don't think this isn't a good argument. many zarr datasets are too big to compute on; should we vendor cubed/dask too?

I think visualizing zarr data is far more basic than doing distributed compute on it. It's very common to use data visualization as a basic sanity check when reading or writing data. And the server is less than 500 lines of code. I have not checked but I suspect this is a bit smaller than dask or cubed.

[psobolewskiPhD]

Very cool! Works quite nicely!
One nice thing that simple-zarr-server offers is a name.

serve(store, host="0.0.0.0", name='data.zarr')  # pass name so reader works

then this works from my CLI:

napari http://URL:8000/data.zarr

(napari readers use extensions)

What's cool is this worked with tifffile as_zarr!

[d-v-b]

Very cool! Works quite nicely! One nice thing that simple-zarr-server offers is a name.

serve(store, host="0.0.0.0", name='data.zarr')  # pass name so reader works

Good idea, I can add this

[psobolewskiPhD]

I did run into one issues, maybe PBCAK, but when the remote file being served was zarr2, if I didn't specify zarr_version=2 in my local (client) zarr.open things didn't work -- i couldn't get the arrays from a group. I happened to know it was zarr2, but in principle that might not be the case?

[d-v-b]

I did run into one issues, maybe PBCAK, but when the remote file being served was zarr2, if I didn't specify zarr_version=2 in my local (client) zarr.open things didn't work -- i couldn't get the arrays from a group. I happened to know it was zarr2, but in principle that might not be the case?

that's a Zarr Python API thing -- we default to Zarr V3. we should have a less ambiguous / better API for this.

[psobolewskiPhD]

Here's my serve script btw:
https://gist.github.com/psobolewskiPhD/c31d73e5e4b3cd7db7ff429fbfca98ee

Also, probably unrelated to this? but I tried, on a lark, to use obstore thinking this is an obstore use case:

from zarr.storage import ObjectStore
from obstore.store import HTTPStore

# Create an obstore HTTP store instance
http_store = HTTPStore(url='host')  # I used my actual hostname here. just obfuscating it

# Wrap it in the Zarr ObjectStore
store = ObjectStore(http_store)

group = zarr.open_group(store, mode='r', zarr_version=2)

Builder errors

GenericError: Generic HTTP error: Error performing GET http://host:9876/.zgroup in 43.958µs - HTTP error: builder error

Debug source:
Generic {
    store: "HTTP",
    source: RetryError(
        RetryErrorImpl {
            method: GET,
            uri: Some(
                http://host:9876/.zgroup,
            ),
            retries: 0,
            max_retries: 10,
            elapsed: 43.958µs,
            retry_timeout: 180s,
            inner: Http(
                HttpError {
                    kind: Unknown,
                    source: reqwest::Error {
                        kind: Builder,
                        source: BadScheme,
                    },
                },
            ),
        },

[psobolewskiPhD]

AH, so I think i conflated things and this http server of course isn't object storage.
https://obspec-utils.readthedocs.io/en/latest/api/aiohttp/
might be more the more correct obstore choice, but that store isn't supported by zarr.

[maxrjones]

but that store isn't supported by zarr.

#3698 would change that, although there are open discussions about how to map obspec to aiohttp - developmentseed/obspec-utils#65.

[maxrjones]

I'm on board with adding this to experimental, especially given the positive responses in this thread/on zulip.

I have a couple questions:

1. We'll find out if something's broken via issues, but how will we find out if people love the feature such that it should be moved into the stable API? Should we include an ExperimentalWarning that links to a discussion where people can comment and advocate for the feature's elevation?
2. Should we keep the functionality in core.keys inside src/zarr/experimental/ for now, until it's used by part of the stable functionality? I'd want it to be as simple as possible to yank this stuff out if we decide not to keep it, so to me it seems helpful to keep experimental internals isolated.