Skip to content

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

46 Commits

Folders and files

Repository files navigation

fair-metadata-integration

Integrate and package all metadata into RO-Crate, linking to other metadata files.

Given a Bilayers workflow specification and the image data an analysis consumed and produced, this package writes a Workflow RO-Crate: an ro-crate-metadata.json describing the run, alongside copies of the files it describes.

Installation

pip install fair-metadata-integration

Requires Python 3.12 or newer. rocrate is installed automatically.

Usage

from fair_metadata_integration import create_ro_crate

create_ro_crate(
    dest_path="crate",
    workflow_schema_filename="config.yaml",
    input_paths=["data/input.ome.zarr"],
    output_paths=["crate/output.ome.zarr"],
    metadata_paths=["data/metadata.yaml"],
    parameters="parameters.yaml",
)
  • dest_path — directory the crate is written to, created if it does not exist.

  • workflow_schema_filename — the Bilayers config.yaml describing the workflow.

  • input_paths — the data the workflow consumed (e.g. OME-Zarr directories or TIFF files). A single input can also be given as just its path.

  • output_paths — the data the workflow produced.

    In input_paths and output_paths, many files can be given as one group: a dict of its name, paths and optionally description, in place of a path. See Groups of files.

  • metadata_paths — further files describing the data (typically text: JSON, YAML, XML, CSV). They are part of the crate, and copied into its root when outside dest_path.

  • parameters — the values the run used: the path of a YAML or JSON file, which is copied into the crate, or a dict, which is written to parameters.yaml in it. The file is about the run; its values are not entities of the crate. They are nested as config.yaml is: an entry with a section_id has its value under that key, so one name can be used in several sections:

    pre_processing:
      normalisation: none
    registration:
      normalisation: true

    Every value is kept; values at no entry of config.yaml are listed in one warning (e.g. registration.tiling).

  • copy_inputs, copy_outputs — copy the inputs, or the outputs, into the crate root when they are outside dest_path (default False: referenced, see below).

  • gen_preview — write an ro-crate-preview.html alongside the metadata (default True).

  • name, description — the name and description of the crate's root dataset; the description defaults to what the crate holds, as RO-Crate requires one.

  • license — the license of the crate's content, which RO-Crate requires: its URL, or a dict of its url, name and description (default: CC BY 4.0). The licenses in COMMON_LICENSES (Creative Commons 4.0 and CC0, Open Data Commons) are named and described by their URL alone.

  • metadata_conforms_to — the schema the metadata files follow, as a dict of its url and optionally its name and version; each metadata file gets a conformsTo link to it.

  • ome_version — when dest_path is itself an OME-Zarr store, the OME-NGFF version it conforms to (default: the version the store states; see below).

  • run_name, start_time, end_time — the name of the run, and when it started and ended (datetimes or ISO text).

  • software — the software implementing the workflow, as a dict of its name and optionally its url and version; the workflow isPartOf it.

workflow_schema_filename is optional: without it, the crate describes just the data — the root dataset with the inputs, outputs and metadata files as its parts — and has no workflow or run. parameters then cannot be given.

OME-Zarr stores, as inputs, outputs or as the crate itself, conform to the OME-NGFF version they state (conformsTo e.g. https://ngff.openmicroscopy.org/0.5/), read from zarr.json (0.5) or .zattrs (0.4 and earlier).

An RO-Crate in an OME-Zarr store

create_zarr_ro_crate describes an OME-Zarr store, written by something else, with an RO-Crate in the store itself. Only ro-crate-metadata.json (the preview, and the list_file of a group of sources) is written; the store is left as it is:

from fair_metadata_integration import create_zarr_ro_crate

create_zarr_ro_crate(
    "cells.ome.zarr",
    metadata_paths=["cells.ome.zarr/acquisition-metadata.yaml"],
    metadata_conforms_to={"url": "https://example.org/model.yaml", "name": "Imaging model", "version": "2.0"},
)

The root dataset is the store: named after it (cells), with the EDAM Zarr format, the OME-NGFF version it conforms to — ome_version, or else the one the store states (a warning if they differ, or if neither is known) — and a description, by default "OME-Zarr image", or "OME-Zarr image with labels". The metadata files must be in the store; they are about the store.

Where the store came from, as the Process Run Crate profile models it:

create_zarr_ro_crate(
    "fused.ome.zarr",
    source_paths=["tiles/tile1.tif", "tiles/tile2.tif"],
    acquisition_metadata=[tile1_metadata, tile2_metadata],
    derivation={"name": "Fusion to OME-Zarr", "end_time": "2026-11-05T16:00:00",
                "software": {"name": "muvis-align", "url": "https://github.com/ccp-volume-em/muvis-align",
                             "version": "1.2"}},
    measurements={"ncc": 0.75, "nmi": {"value": 0.5, "description": "Registration NMI"}},
)
  • source_paths — the images the store was made from: the store isBasedOn them, referenced by their file:// URI and not copied into the store; like any file or folder the crate describes, they are among its root's hasPart. Many images can be one group, as for input_paths.
  • acquisition_metadata — metadata of (some of) the sources, as dicts: an Image acquisition CreateAction (#acquisition) with the sources as result, an IndividualProduct instrument from the manufacturer, model and serial number found in it (preferring fields under an instrument, microscope or device), and the earliest and latest acquisition date as start and end time. Keys are matched ignoring case and separators (SerialNumber, serial_number).
  • derivation — how the store was made: a CreateAction (#derivation) with the sources as object, the store as result and the software as instrument; its end time is the store's datePublished.
  • measurements — values measured on the store, as PropertyValues of its variableMeasured: a value, or a dict of its value, description and unit.

Helpers for writing a run's files

To package a run of a tool with no hand-written Bilayers config, generate one from the values the run used, and write those values as its parameters file:

from fair_metadata_integration import (WORKFLOW_CONFIG_NAME, bilayers_config, bilayers_format,
                                       create_ro_crate, image_entry, write_bilayers_config,
                                       write_parameters)

values = {"model": "MitoNet_v1", "confidence": 0.5}
config = bilayers_config(
    "empanada_3d", values,
    inputs=[image_entry("input_image", formats=[bilayers_format("data/input.tif")])],
    outputs=[image_entry("segmentation", formats=["omezarr"], section="outputs")],
    citations=[{"name": "empanada-napari", "doi": "10.1083/jcb.202208005"}],
    docker_image={"org": "ghcr.io/me", "name": "tool", "tag": "1.0", "platform": "linux/amd64"},
    cli_command="python run.py",
    choices={"model": ["MitoNet_v1", "MitoNet_v1_mini"]},
)
config_path = write_bilayers_config(f"crate/{WORKFLOW_CONFIG_NAME}", config)
parameters_path = write_parameters("crate", values)  # crate/parameters.yaml
create_ro_crate("crate", config_path, ["data/input.tif"], ["crate/output.ome.zarr"],
                parameters=parameters_path)
  • bilayers_config makes every value a parameter whose default is that value, typed from it (checkbox, integer, float, textbox, or dropdown when it has choices); values that are None are left out. With section, the parameters get that section_id, so the run's values go under that key in parameters.yaml. display_only takes entries for values that are shown but not passed to the command, e.g. those of other tools of the run, made with display_entry(name, value, section=...). parameter_entry and image_entry build single entries.
  • write_bilayers_config validates the config by converting it with biomero-schema's Bilayers adapter when biomero-schema is installed, and writes it.
  • write_parameters writes possibly nested values to parameters.yaml (PARAMETERS_FILE_NAME), with values YAML cannot hold (e.g. paths) as text; plain_value and write_yaml do this for any data.

Command line

The same is available as the create-ro-crate command (or python -m fair_metadata_integration):

create-ro-crate crate config.yaml \
    --input data/input.ome.zarr \
    --output crate/output.ome.zarr \
    --metadata data/metadata.yaml \
    --parameters parameters.yaml

--input, --output and --metadata each take one or more paths; --copy-inputs, --copy-outputs, --run-name, --start-time, --end-time, --software NAME [URL [VERSION]] and --gen-preview / --no-gen-preview match the arguments above. Run create-ro-crate --help for details.

Only the workflow specification, the parameters file and metadata files are copied into the crate. Data inside dest_path, such as the outputs the workflow wrote there, is part of the crate where it is. Data outside it, typically the original input, is not copied but referenced by its file:// URI (unless copy_inputs or copy_outputs is set):

crate/
├── config.yaml
├── output.ome.zarr/
├── ro-crate-metadata.json
└── ro-crate-preview.html

ro-crate-preview.html is the human-readable RO-Crate Website for the crate: a static HTML page (no scripts) with a section for every entity and all its properties. References between entities link to each other's section, so you can follow e.g. the run to the workflow and on to the files it produced; external URIs, types and property names link to their definitions. Files and folders also get a summary of their content:

  • CSV and TSV files: number of rows and columns, and the column names (with their types, for CellProfiler's # header line).
  • The Bilayers config.yaml: citations, Docker image, command, and the inputs, outputs and parameters, with the values the run gave the parameters (read from the parameters file).
  • OME-Zarr folders (Zarr v2 and v3): axes and units, the shape, chunks, data type and scale of each resolution level, channels, label images, and the plate layout or bioformats2raw image series.

Text files, which a browser would download rather than show, link to a browsable view in ro-crate-preview_files/: CSV and TSV files as a table (the first 1,000 rows), JSON and YAML files as a tree of keys and values you can collapse, and other text files such as CellProfiler pipelines as text (the first 1,000,000 characters).

The preview is described in the metadata as a CreativeWork about the root dataset, and is not part of the root dataset's hasPart.

What the crate contains

The workflow specification is the crate's mainEntity, typed as a ComputationalWorkflow. The metadata descriptor declares RO-Crate 1.3, and the root dataset declares the Workflow RO-Crate 1.1 profile, and the Process Run Crate and Workflow Run Crate 0.6 profiles for the run (not shown): the versions built on RO-Crate 1.3. Crates pass the RO-Crate validator's required checks for these profiles:

{
  "@id": "ro-crate-metadata.json",
  "@type": "CreativeWork",
  "about": { "@id": "./" },
  "conformsTo": { "@id": "https://w3id.org/ro/crate/1.3" }
},
{
  "@id": "./",
  "@type": "Dataset",
  "conformsTo": [
    { "@id": "https://w3id.org/workflowhub/workflow-ro-crate/1.1" }
  ],
  "mainEntity": { "@id": "config.yaml" }
}

The workflow's programmingLanguage is a ComputerLanguage entity for the Bilayers specification, identified by its Zenodo concept DOI 10.5281/zenodo.17652333, which resolves to the latest release of the spec, with the version of the release the configs follow (1.0.0rc1). Software (e.g. the software of the run) gives its version.

Folders (e.g. OME-Zarr) are Dataset entities, files File entities, with a path relative to the crate when they are in it and their file:// URI when they are not. Their encodingFormat is an EDAM format, from the file name: format_3915 for Zarr, format_3727 for OME-TIFF, format_3591 for TIFF, and PNG, JPEG, HDF5, CSV, TSV, xlsx, JSON, YAML, XML and text. Files also give their IANA media type, and each EDAM format is a WebPage contextual entity with its name, as RO-Crate recommends.

The run is a CreateAction, mentioned by the root dataset, with the workflow as instrument, the input data as object and the output data as result. The workflow's parameters and the run's values are not entities of the crate: they are in config.yaml and the parameters file, which is about the run:

{
  "@id": "#run",
  "@type": "CreateAction",
  "name": "Run of config.yaml",
  "instrument": { "@id": "config.yaml" },
  "object": [{ "@id": "file:///data/input.ome.zarr/" }],
  "result": [{ "@id": "output.ome.zarr/" }]
},
{
  "@id": "parameters.yaml",
  "@type": "File",
  "about": { "@id": "#run" }
}

Groups of files

A run over many files, e.g. the tiles of a mosaic, can give them as one group:

create_ro_crate("crate", "config.yaml",
                input_paths={"name": "Source tiles", "description": "Tiles of the mosaic",
                             "paths": ["data/tiles/tile1.tif", "data/tiles/tile2.tif"]})

The group is a Dataset with the files as its hasPart; it is what the run's object (or result, or a store's isBasedOn) refers to. Each file is still its own entity. The Dataset is the files' common folder (its path in the crate, or its file:// URI), listing only the files given, not everything in it. When that folder would be the crate itself (files in or copied to its root), the Dataset has a local id, #dataset- and its name.

For hundreds of files, give the group a list_file name: the files are then not entities of the crate. Their paths, relative to the group's folder, are written to that CSV file in the crate (one path column), which is about the group, and the group has no hasPart. A listed group is referenced, so it cannot be copied (copy_inputs / copy_outputs).

{
  "@id": "file:///data/tiles/",
  "@type": "Dataset",
  "name": "Source tiles"
},
{
  "@id": "tiles.csv",
  "@type": "File",
  "about": { "@id": "file:///data/tiles/" }
}

Without list_file:

{
  "@id": "file:///data/tiles/",
  "@type": "Dataset",
  "name": "Source tiles",
  "description": "Tiles of the mosaic",
  "hasPart": [{ "@id": "file:///data/tiles/tile1.tif" }, { "@id": "file:///data/tiles/tile2.tif" }]
}

Filled in automatically

What RO-Crate and the run profiles recommend is filled in from what is known, so a crate passes the validator's required checks and nearly all recommended ones:

  • every file and folder: its name, a description from its format (e.g. "OME-Zarr image", "CSV file") and, for a file, its contentSize;
  • the run: a description, and times with their UTC offset (a datetime is local time; text, such as an acquisition time, is kept as it is);
  • the workflow: its format, size, a description from the config's first citation, conformsTo the Bioschemas ComputationalWorkflow profile, and the version and url of the software it is part of;
  • a README.md about the crate, written from its name and description; a README of your own is described as it is, never overwritten;
  • a property with one value is written as that value, not a list of one.

The run's agent and the crate's publisher, a person or organisation, are not known, so they are left out. A Bilayers config can offer the licenses as a drop-down with license_entry().

The crate's JSON-LD context is extended with terms for describing biological imaging metadata — biosample, specimen, organism_classification, preparation_method, channel and others — so those properties can be added to the crate's entities.

Development

git clone https://github.com/NL-BioImaging/fair-metadata-integration.git
cd fair-metadata-integration
pip install -e ".[test]"
pytest

To inspect an example crate (OME-Zarr input, OME-Zarr and CellProfiler CSV outputs), test_example_crate writes one and prints its path and that of its preview; pytest keeps the folders of its last three runs. Run the test module as a script to write it to a folder of your choice instead:

pytest -s -k test_example_crate
python tests/test_metadata_integration.py [folder]

Releasing

The package version comes from the git tag, via setuptools-scm; there is no version to bump in pyproject.toml.

  1. Optionally, do a dry run: run the Publish to PyPI workflow manually from the Actions tab. It builds, tests and uploads a development version (e.g. 0.2.4.dev2) to TestPyPI.
  2. Create a GitHub release with a new tag, e.g. v0.2.4, on the commit to release. Publishing the release builds, tests and uploads that version to PyPI.

PyPI never accepts the same version twice. If a release fails after uploading, release a new version rather than reusing the tag.

Use of AI

Generative AI coding assistants were used in developing this software, including tests and documentation. All changes are directed and approved by a maintainer, and checked by automated tests where applicable.

License

MIT

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages