Integrate and package all metadata into RO-Crate, linking to other metadata files.
Given a Bilayers workflow
specification and the image data an analysis consumed and produced, this package
writes a Workflow RO-Crate:
an ro-crate-metadata.json describing the run, alongside copies of the files it
describes.
pip install fair-metadata-integrationRequires Python 3.12 or newer. rocrate is
installed automatically.
from fair_metadata_integration import create_ro_crate
create_ro_crate(
dest_path="crate",
workflow_schema_filename="config.yaml",
input_paths=["data/input.ome.zarr"],
output_paths=["crate/output.ome.zarr"],
metadata_paths=["data/metadata.yaml"],
parameters="parameters.yaml",
)-
dest_path— directory the crate is written to, created if it does not exist. -
workflow_schema_filename— the Bilayersconfig.yamldescribing the workflow. -
input_paths— the data the workflow consumed (e.g. OME-Zarr directories or TIFF files). A single input can also be given as just its path. -
output_paths— the data the workflow produced.In
input_pathsandoutput_paths, many files can be given as one group: adictof itsname,pathsand optionallydescription, in place of a path. See Groups of files. -
metadata_paths— further files describing the data (typically text: JSON, YAML, XML, CSV). They are part of the crate, and copied into its root when outsidedest_path. -
parameters— the values the run used: the path of a YAML or JSON file, which is copied into the crate, or adict, which is written toparameters.yamlin it. The file isaboutthe run; its values are not entities of the crate. They are nested asconfig.yamlis: an entry with asection_idhas its value under that key, so one name can be used in several sections:pre_processing: normalisation: none registration: normalisation: true
Every value is kept; values at no entry of
config.yamlare listed in one warning (e.g.registration.tiling). -
copy_inputs,copy_outputs— copy the inputs, or the outputs, into the crate root when they are outsidedest_path(defaultFalse: referenced, see below). -
gen_preview— write anro-crate-preview.htmlalongside the metadata (defaultTrue). -
name,description— the name and description of the crate's root dataset; the description defaults to what the crate holds, as RO-Crate requires one. -
license— the license of the crate's content, which RO-Crate requires: its URL, or adictof itsurl,nameanddescription(default: CC BY 4.0). The licenses inCOMMON_LICENSES(Creative Commons 4.0 and CC0, Open Data Commons) are named and described by their URL alone. -
metadata_conforms_to— the schema the metadata files follow, as adictof itsurland optionally itsnameandversion; each metadata file gets aconformsTolink to it. -
ome_version— whendest_pathis itself an OME-Zarr store, the OME-NGFF version it conforms to (default: the version the store states; see below). -
run_name,start_time,end_time— the name of the run, and when it started and ended (datetimes or ISO text). -
software— the software implementing the workflow, as adictof itsnameand optionally itsurlandversion; the workflowisPartOfit.
workflow_schema_filename is optional: without it, the crate describes just the
data — the root dataset with the inputs, outputs and metadata files as its
parts — and has no workflow or run. parameters then cannot be given.
OME-Zarr stores, as inputs, outputs or as the crate itself, conform to the
OME-NGFF version they state (conformsTo e.g.
https://ngff.openmicroscopy.org/0.5/), read from zarr.json (0.5) or
.zattrs (0.4 and earlier).
create_zarr_ro_crate describes an OME-Zarr store, written by something else,
with an RO-Crate in the store itself. Only ro-crate-metadata.json (the
preview, and the list_file of a group of sources) is written; the store is left
as it is:
from fair_metadata_integration import create_zarr_ro_crate
create_zarr_ro_crate(
"cells.ome.zarr",
metadata_paths=["cells.ome.zarr/acquisition-metadata.yaml"],
metadata_conforms_to={"url": "https://example.org/model.yaml", "name": "Imaging model", "version": "2.0"},
)The root dataset is the store: named after it (cells), with the EDAM Zarr
format, the OME-NGFF version it conforms to — ome_version, or else the one the
store states (a warning if they differ, or if neither is known) — and a
description, by default "OME-Zarr image", or "OME-Zarr image with labels".
The metadata files must be in the store; they are about the store.
Where the store came from, as the Process Run Crate profile models it:
create_zarr_ro_crate(
"fused.ome.zarr",
source_paths=["tiles/tile1.tif", "tiles/tile2.tif"],
acquisition_metadata=[tile1_metadata, tile2_metadata],
derivation={"name": "Fusion to OME-Zarr", "end_time": "2026-11-05T16:00:00",
"software": {"name": "muvis-align", "url": "https://github.com/ccp-volume-em/muvis-align",
"version": "1.2"}},
measurements={"ncc": 0.75, "nmi": {"value": 0.5, "description": "Registration NMI"}},
)source_paths— the images the store was made from: the storeisBasedOnthem, referenced by theirfile://URI and not copied into the store; like any file or folder the crate describes, they are among its root'shasPart. Many images can be one group, as forinput_paths.acquisition_metadata— metadata of (some of) the sources, asdicts: anImage acquisitionCreateAction(#acquisition) with the sources asresult, anIndividualProductinstrument from the manufacturer, model and serial number found in it (preferring fields under an instrument, microscope or device), and the earliest and latest acquisition date as start and end time. Keys are matched ignoring case and separators (SerialNumber,serial_number).derivation— how the store was made: aCreateAction(#derivation) with the sources asobject, the store asresultand the software asinstrument; its end time is the store'sdatePublished.measurements— values measured on the store, asPropertyValues of itsvariableMeasured: a value, or adictof itsvalue,descriptionandunit.
To package a run of a tool with no hand-written Bilayers config, generate one from the values the run used, and write those values as its parameters file:
from fair_metadata_integration import (WORKFLOW_CONFIG_NAME, bilayers_config, bilayers_format,
create_ro_crate, image_entry, write_bilayers_config,
write_parameters)
values = {"model": "MitoNet_v1", "confidence": 0.5}
config = bilayers_config(
"empanada_3d", values,
inputs=[image_entry("input_image", formats=[bilayers_format("data/input.tif")])],
outputs=[image_entry("segmentation", formats=["omezarr"], section="outputs")],
citations=[{"name": "empanada-napari", "doi": "10.1083/jcb.202208005"}],
docker_image={"org": "ghcr.io/me", "name": "tool", "tag": "1.0", "platform": "linux/amd64"},
cli_command="python run.py",
choices={"model": ["MitoNet_v1", "MitoNet_v1_mini"]},
)
config_path = write_bilayers_config(f"crate/{WORKFLOW_CONFIG_NAME}", config)
parameters_path = write_parameters("crate", values) # crate/parameters.yaml
create_ro_crate("crate", config_path, ["data/input.tif"], ["crate/output.ome.zarr"],
parameters=parameters_path)bilayers_configmakes every value a parameter whose default is that value, typed from it (checkbox,integer,float,textbox, ordropdownwhen it haschoices); values that areNoneare left out. Withsection, the parameters get thatsection_id, so the run's values go under that key inparameters.yaml.display_onlytakes entries for values that are shown but not passed to the command, e.g. those of other tools of the run, made withdisplay_entry(name, value, section=...).parameter_entryandimage_entrybuild single entries.write_bilayers_configvalidates the config by converting it with biomero-schema's Bilayers adapter when biomero-schema is installed, and writes it.write_parameterswrites possibly nested values toparameters.yaml(PARAMETERS_FILE_NAME), with values YAML cannot hold (e.g. paths) as text;plain_valueandwrite_yamldo this for any data.
The same is available as the create-ro-crate command (or
python -m fair_metadata_integration):
create-ro-crate crate config.yaml \
--input data/input.ome.zarr \
--output crate/output.ome.zarr \
--metadata data/metadata.yaml \
--parameters parameters.yaml--input, --output and --metadata each take one or more paths;
--copy-inputs, --copy-outputs, --run-name, --start-time, --end-time,
--software NAME [URL [VERSION]] and --gen-preview / --no-gen-preview match
the arguments above.
Run create-ro-crate --help for details.
Only the workflow specification, the parameters file and metadata files are
copied into the crate. Data inside dest_path, such as the outputs the workflow
wrote there, is part of the crate where it is. Data outside it, typically the
original input, is not copied but referenced by its file:// URI (unless
copy_inputs or copy_outputs is set):
crate/
├── config.yaml
├── output.ome.zarr/
├── ro-crate-metadata.json
└── ro-crate-preview.html
ro-crate-preview.html is the human-readable
RO-Crate Website
for the crate: a static HTML page (no scripts) with a section for every entity
and all its properties. References between entities link to each other's
section, so you can follow e.g. the run to the workflow and on to the files it
produced; external URIs, types and property names link to their
definitions. Files and folders also get a summary of their content:
- CSV and TSV files: number of rows and columns, and the column names (with
their types, for CellProfiler's
# headerline). - The Bilayers
config.yaml: citations, Docker image, command, and the inputs, outputs and parameters, with the values the run gave the parameters (read from the parameters file). - OME-Zarr folders (Zarr v2 and v3): axes and units, the shape, chunks, data type and scale of each resolution level, channels, label images, and the plate layout or bioformats2raw image series.
Text files, which a browser would download rather than show, link to a
browsable view in ro-crate-preview_files/: CSV and TSV files as a table (the
first 1,000 rows), JSON and YAML files as a tree of keys and values you can
collapse, and other text files such as CellProfiler pipelines as text (the first
1,000,000 characters).
The preview is described in the metadata as a CreativeWork about the root
dataset, and is not part of the root dataset's hasPart.
The workflow specification is the crate's mainEntity, typed as a
ComputationalWorkflow. The metadata descriptor declares RO-Crate 1.3, and the
root dataset declares the Workflow RO-Crate 1.1 profile, and the
Process Run Crate and Workflow Run Crate
0.6 profiles for the run (not shown): the versions built on RO-Crate 1.3. Crates
pass the RO-Crate validator's
required checks for these profiles:
{
"@id": "ro-crate-metadata.json",
"@type": "CreativeWork",
"about": { "@id": "./" },
"conformsTo": { "@id": "https://w3id.org/ro/crate/1.3" }
},
{
"@id": "./",
"@type": "Dataset",
"conformsTo": [
{ "@id": "https://w3id.org/workflowhub/workflow-ro-crate/1.1" }
],
"mainEntity": { "@id": "config.yaml" }
}The workflow's programmingLanguage is a ComputerLanguage entity for the
Bilayers specification, identified by its Zenodo concept DOI
10.5281/zenodo.17652333, which
resolves to the latest release of the spec, with the version of the release
the configs follow (1.0.0rc1). Software (e.g. the software of the run) gives
its version.
Folders (e.g. OME-Zarr) are Dataset entities, files File entities, with a
path relative to the crate when they are in it and their file:// URI when
they are not. Their encodingFormat is an EDAM
format, from the file name: format_3915 for Zarr, format_3727 for OME-TIFF,
format_3591 for TIFF, and PNG, JPEG, HDF5, CSV, TSV, xlsx, JSON, YAML, XML and
text. Files also give their IANA media type, and each EDAM format is a WebPage
contextual entity with its name, as RO-Crate recommends.
The run is a CreateAction, mentioned by the root dataset, with the workflow
as instrument, the input data as object and the output data as result. The
workflow's parameters and the run's values are not entities of the crate: they
are in config.yaml and the parameters file, which is about the run:
{
"@id": "#run",
"@type": "CreateAction",
"name": "Run of config.yaml",
"instrument": { "@id": "config.yaml" },
"object": [{ "@id": "file:///data/input.ome.zarr/" }],
"result": [{ "@id": "output.ome.zarr/" }]
},
{
"@id": "parameters.yaml",
"@type": "File",
"about": { "@id": "#run" }
}A run over many files, e.g. the tiles of a mosaic, can give them as one group:
create_ro_crate("crate", "config.yaml",
input_paths={"name": "Source tiles", "description": "Tiles of the mosaic",
"paths": ["data/tiles/tile1.tif", "data/tiles/tile2.tif"]})The group is a Dataset with the files as its hasPart; it is what the run's
object (or result, or a store's isBasedOn) refers to. Each file is still
its own entity. The Dataset is the files' common folder (its path in the
crate, or its file:// URI), listing only the files given, not everything in
it. When that folder would be the crate itself (files in or copied to its root),
the Dataset has a local id, #dataset- and its name.
For hundreds of files, give the group a list_file name: the files are then not
entities of the crate. Their paths, relative to the group's folder, are written
to that CSV file in the crate (one path column), which is about the group, and
the group has no hasPart. A listed group is referenced, so it cannot be copied
(copy_inputs / copy_outputs).
{
"@id": "file:///data/tiles/",
"@type": "Dataset",
"name": "Source tiles"
},
{
"@id": "tiles.csv",
"@type": "File",
"about": { "@id": "file:///data/tiles/" }
}Without list_file:
{
"@id": "file:///data/tiles/",
"@type": "Dataset",
"name": "Source tiles",
"description": "Tiles of the mosaic",
"hasPart": [{ "@id": "file:///data/tiles/tile1.tif" }, { "@id": "file:///data/tiles/tile2.tif" }]
}What RO-Crate and the run profiles recommend is filled in from what is known, so a crate passes the validator's required checks and nearly all recommended ones:
- every file and folder: its
name, adescriptionfrom its format (e.g. "OME-Zarr image", "CSV file") and, for a file, itscontentSize; - the run: a
description, and times with their UTC offset (adatetimeis local time; text, such as an acquisition time, is kept as it is); - the workflow: its format, size, a
descriptionfrom the config's first citation,conformsTothe Bioschemas ComputationalWorkflow profile, and theversionandurlof thesoftwareit is part of; - a
README.mdabout the crate, written from its name and description; a README of your own is described as it is, never overwritten; - a property with one value is written as that value, not a list of one.
The run's agent and the crate's publisher, a person or organisation, are
not known, so they are left out. A Bilayers config can offer the licenses as a
drop-down with license_entry().
The crate's JSON-LD context is extended with terms for describing biological
imaging metadata — biosample, specimen, organism_classification,
preparation_method, channel and others — so those properties can be added to
the crate's entities.
git clone https://github.com/NL-BioImaging/fair-metadata-integration.git
cd fair-metadata-integration
pip install -e ".[test]"
pytestTo inspect an example crate (OME-Zarr input, OME-Zarr and CellProfiler CSV
outputs), test_example_crate writes one and prints its path and that of its
preview; pytest keeps the folders of its last three runs. Run the test module as
a script to write it to a folder of your choice instead:
pytest -s -k test_example_crate
python tests/test_metadata_integration.py [folder]The package version comes from the git tag, via
setuptools-scm; there is no version
to bump in pyproject.toml.
- Optionally, do a dry run: run the Publish to PyPI workflow manually from
the Actions tab. It builds, tests and uploads a development version (e.g.
0.2.4.dev2) to TestPyPI. - Create a GitHub release with a new tag, e.g.
v0.2.4, on the commit to release. Publishing the release builds, tests and uploads that version to PyPI.
PyPI never accepts the same version twice. If a release fails after uploading, release a new version rather than reusing the tag.
Generative AI coding assistants were used in developing this software, including tests and documentation. All changes are directed and approved by a maintainer, and checked by automated tests where applicable.