Skip to content

Import Zarr contracts from biomero-schema and own the workflow color - #88

Open
TorecLuik wants to merge 17 commits into
masterfrom
provenance-views-cleanup
Open

TorecLuik wants to merge 17 commits into
masterfrom
provenance-views-cleanup

Conversation

@TorecLuik

@TorecLuik TorecLuik commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #87
Refs NL-BioImaging/NL-BIOMERO#93

Scripts side of the core clean-up in NL-BioImaging/biomero (provenance views, projections and schema re-exports).

Changes

  • _SLURM_Image_Transfer.py and SLURM_Run_Workflow.py import the Zarr contracts from biomero_schema.zarr (core's biomero.zarr_contracts re-export is removed).
  • SLURM_Get_Results.py, SLURM_Import_Results.py and SLURM_Init_environment.py import from biomero.metadata_views (renamed from biomero.provenance; core removes the old name, which only existed in prereleases). README updated.
  • The two result scripts format a reduced (too large for the OMERO index) value with core's format_large_field_reference, so core recognises exactly that text when it refreshes the view and keeps it reduced. The text itself is unchanged.
  • The Blueprint workflow ROI colour (QUALITATIVE_COLOR_SCHEME, resolve_workflow_color) moves from biomero.constants to SLURM_Run_Workflow.py, which already owns the canonical import parameters and forwards the resolved colour (ROI_Color) to the result scripts.

Minimum core release

These scripts need the matching core release, which contains NL-BioImaging/biomero#104 (it provides biomero.metadata_views and format_large_field_reference, and no longer has biomero.provenance, biomero.zarr_contracts or constants.resolve_workflow_color). Scripts and core are released together, so no minimum version is stated: the README names the renamed biomero.metadata_views, and the three import error messages say that the matching release is needed.

Conflict with #97 (merges first): the gated import block in SLURM_Run_Workflow.py, resolved in this branch to from biomero_schema.zarr import CanonicalInput, TRANSFER_INPUT_MARKER.

Release notes: Scripts import the Zarr contracts from biomero_schema.zarr and the metadata views from biomero.metadata_views; the workflow ROI colour is chosen by SLURM_Run_Workflow.py. Requires the matching biomero release.

Behaviour change

  • SLURM_Get_Results.py / SLURM_Import_Results.py use the forwarded ROI_Color as given. When a result script is run on its own with an empty colour, it no longer derives a per-workflow colour; Labels2Rois then applies its own default (empty means no explicit colour). Runs through SLURM_Run_Workflow.py are unchanged (it always forwards the resolved colour when ROIs are requested). Explicit invalid colours are still rejected (by the runner and by Labels2Rois).

Merge order (production and test PRs, interleaved)

Each step keeps master CI green: every (production state, test-suite state) pair on this path was run locally with core 2.9.0b26 from PyPI (482, 484, 484, 513, 524, 588, 594, 597, 606 passed; no failures). The production PRs merge cleanly in this order: each branch already contains the conflict resolutions with the PRs before it.

  1. test PR Tests: install biomero core 2.9.0b26 in the harness #103 (test-suite: biomero>=2.9.0b26,<3 in tests/requirements.txt, so core-dependent tests run in CI)
  2. test PRs Tests: schema imports, metadata view imports and workflow color #89, Tests: detached resume, cancel and registration #91 and Tests: per-directory canonical sidecars and unwritable folders #99 (capability-gated; they pass for the current master and for the new source)
  3. production PR Remove the input marker from results after import on every path #97
  4. test PR Tests: remove the input marker from results after import #98 (fails on master before Remove the input marker from results after import on every path #97, by design)
  5. production PR Import Zarr contracts from biomero-schema and own the workflow color #88 (this PR)
  6. production PR Detached processing: own launcher registration, resume and cancel in the scripts #90
  7. production PR Migration script: move canonical sidecars via importer helpers #95
  8. test PR Tests: shared output names come from core constants #102
  9. production PR Use the core script-output constants #101

A production PR's own check stays red until the test PRs before it in this list are merged (CI runs the shared harness from test-suite, which is not pinned); merge the test PR, then rerun the check.

🤖 Generated with Claude Code

TorecLuik and others added 2 commits November 12, 2026 14:28
- Import the Zarr contracts from biomero_schema.zarr; biomero.zarr_contracts
  is removed from core.
- Import the metadata view functions from biomero.metadata_views (renamed from
  biomero.provenance) and format reduced values with core's
  format_large_field_reference, so core recognises them on refresh.
- Move the Blueprint workflow color to SLURM_Run_Workflow.py, which forwards
  the resolved color to the result scripts.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- Fail with a clear message when the installed biomero predates
  biomero.metadata_views; README states the minimum core release.
- Move the colour palette to the module constants area of the runner.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@TorecLuik

Copy link
Copy Markdown
Collaborator Author

Independent review summary

Reviewer: Sonnet 5.5 subagent that did not write the change (read-only, 68 mutation checks across the three PRs). Verdict: no blockers.

# Severity Finding Resolution
S1 should fix Merge-order text named the wrong repository Fixed: test-suite PR #89, then biomero#104, then this PR.
S2 should fix Minimum core release not stated; import failure not legible Stated in the description, release notes and README; the three scripts that import biomero.metadata_views raise an ImportError explaining that a matching biomero release is required.
S3 should fix Behaviour not covered by tests here Covered in #89 (scripts tests live on the test-suite branch).
N nit Colour constants not in the constants area Moved next to the other module constants of SLURM_Run_Workflow.py.

Harness run locally against this branch with the updated #89 tests: 493 passed, 7 skipped. This PR's own CI stays red until #89 is merged (shared harness).

TorecLuik and others added 4 commits November 13, 2026 14:02
The workflow runner now owns the cleanup of .biomero-input.json: after the
result import succeeded it removes the marker from the imported result folder.
SLURM_Import_Results.py reports that folder as the Results_Path output.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
biomero.zarr_contracts is being removed from core; the marker constant is
defined in biomero_schema.zarr, which the scripts already import.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ogether

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The harness still expects the CanonicalInput import from
biomero.zarr_contracts until #88 and its tests merge; only the new
TRANSFER_INPUT_MARKER import comes from biomero_schema.zarr.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TorecLuik and others added 2 commits November 13, 2026 14:30
Accept only .analyzed/<workflow>/<timestamp>, never fail the run on a cleanup
error, warn on unreadable folders, drop unreachable parameters, name the
shallow storage condition and document the Results_Path output.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Both scripts take RESULTS_PATH_OUTPUT from core (biomero#110), like
WORKFLOW_UUID_OUTPUT, instead of defining it locally.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Resolve the gated import block with #97 (the input marker cleanup, which
merges first) to
`from biomero_schema.zarr import CanonicalInput, TRANSFER_INPUT_MARKER`.
Name the released minimum core version (2.9.0b26) in the README and in the
import error messages.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TorecLuik added a commit that referenced this pull request Nov 13, 2026
#97 and #88 merge before this PR; merging them here applies cleanly.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TorecLuik added a commit that referenced this pull request Nov 13, 2026
…er>=1.5.0b14

The merge is clean. The scripts declared no importer minimum anywhere, so
the README now names biomero-importer>=1.5.0b14 (it provides the
PROCESSED_DATA_FOLDER and sidecar helpers this PR imports).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TorecLuik added a commit that referenced this pull request Nov 13, 2026
Conflicts with the PRs that merge before this one, resolved by taking their
code and applying the same mechanical replacement of output-name literals
with biomero.constants.results.*_OUTPUT:
- SLURM_Run_Workflow.py: keep RESULTS_FOLDER_NAME and ZARR_ROOT_FILES from #97,
  drop the local CANONICAL_INPUTS_OUTPUT; the export, conversion and import
  stages of #90 use MESSAGE_OUTPUT, URL_OUTPUT and FILE_ANNOTATION_OUTPUT.
- SLURM_Run_Workflow_Batched.py: use #90's register_detached_batches and
  MESSAGE_OUTPUT for the launcher message.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TorecLuik added a commit that referenced this pull request Nov 13, 2026
The merge is clean.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TorecLuik added a commit that referenced this pull request Nov 13, 2026
The input marker cleanup (#97) merges before the Zarr contract move (#88) and
already imports its marker name from biomero_schema.zarr next to the
remaining biomero.zarr_contracts import. Count a script with any old-form
import as the old form, so the harness passes for source with and without
that cleanup.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@TorecLuik
TorecLuik marked this pull request as ready for review November 13, 2026 15:53
TorecLuik added a commit that referenced this pull request Nov 16, 2026
* Tests: schema imports, metadata view imports and workflow color in the runner

Capability-gated: they follow the source revision that imports
biomero_schema.zarr and biomero.metadata_views, and skip or fall back for
older revisions.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Tests: wrap-around colours, forwarding, literal reduced text, strict gates

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Accept the partly migrated import block of the input marker cleanup

The input marker cleanup (#97) merges before the Zarr contract move (#88) and
already imports its marker name from biomero_schema.zarr next to the
remaining biomero.zarr_contracts import. Count a script with any old-form
import as the old form, so the harness passes for source with and without
that cleanup.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
TorecLuik and others added 2 commits November 16, 2026 14:23
…e stores

The result folder is the ANALYZED_DATA_FOLDER setting (read_analyzed_data_folder),
not a local constant. The marker sits at the root of each exported store, one
level below the result folder, so the sweep no longer detects Zarr stores.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Scripts and core are released together, so the README and the import
error name the renamed module, not a version floor.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TorecLuik added a commit that referenced this pull request Nov 16, 2026
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@TorecLuik

Copy link
Copy Markdown
Collaborator Author

Independent review

Scope: only the forward-merge commit 8425807 ("Merge #97 into provenance-views-cleanup; state no minimum core version"), compared with both parents (9c4344c and #97 head dfc2243) via git show --remerge-diff. The PR's original content was not re-reviewed.

Verdict: the merge is correct. No blocking findings.

  • The only conflict is the import block of _data/SLURM_Import_Results.py. It is resolved correctly: it keeps from biomero.storage import read_analyzed_data_folder from Remove the input marker from results after import on every path #97 and the try: from biomero.metadata_views import ... guard from this PR, and drops Remove the input marker from results after import on every path #97's stale from biomero.provenance import render_workflow_metadata.
  • No conflict markers remain; the five touched scripts parse; no duplicate top-level definitions; SLURM_Run_Workflow.py still has the Path, read_analyzed_data_folder and constants.results.RESULTS_PATH_OUTPUT uses from Remove the input marker from results after import on every path #97.
  • No version floor is stated any more. git grep 'biomero>=' finds nothing in the tree. The README sentence now says the release "requires core's biomero.metadata_views ... and biomero.maintenance APIs"; the import errors in SLURM_Get_Results.py, SLURM_Import_Results.py and admin/SLURM_Init_environment.py say "the biomero release that matches them". No 2.9.0b25..b29 string remains.
Severity Where Finding
nit SLURM_Import_Results.py:121 (error text) The message still names biomero.provenance as the former module name; fine, as it explains the rename. No change needed.

🤖 Generated with Claude Code

TorecLuik and others added 2 commits November 16, 2026 14:55
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@TorecLuik

Copy link
Copy Markdown
Collaborator Author

Resolved in 5714bb4: #97 (866958e) merged in, clean. The biomero.provenance mention in the error text stays as it explains the rename.

🤖 Generated with Claude Code

TorecLuik and others added 2 commits November 16, 2026 16:03
…the folder name

Core will not provide an ANALYZED_DATA_FOLDER setting for 1.8 (tracked in
NL-BIOMERO#111). SLURM_Import_Results.py keeps one RESULTS_FOLDER constant,
and the marker sweep accepts only a result folder of this run's workflow:
<...>/<workflow id>/<timestamp>. No general constants remain in the run script.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TorecLuik added a commit that referenced this pull request Nov 16, 2026
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@TorecLuik

Copy link
Copy Markdown
Collaborator Author

Independent review

Reviewer: independent agent. Scope: the latest forward-merge commit of this branch (329487a) only (conflict resolution), plus a repository-wide search; the PR's own change was reviewed in its original review.

Verdict: approve the forward merge. No blocking, no should fix.

  • Merge commit checked with git show --cc: merge of Remove the input marker from results after import on every path #97 into this branch. The one conflict, the import block of SLURM_Import_Results.py, is resolved by keeping the biomero.metadata_views import (with its clear ImportError) and dropping from biomero.storage import read_analyzed_data_folder.
  • git grep over the branch head finds no biomero.storage, ANALYZED_DATA_FOLDER, read_analyzed_data_folder or analyzed_data_folder. RESULTS_FOLDER = ".analyzed" is the only code use of the folder name in SLURM_Import_Results.py (the remaining .analyzed text is in docstrings and comments).
  • SLURM_Run_Workflow.py and SLURM_Import_Results.py parse at this head.
  • Merge order from Remove the input marker from results after import on every path #97's description still holds: each branch contains the one before it, and no duplicate definitions or changed signatures were introduced by the resolution.

🤖 Generated with Claude Code

TorecLuik and others added 2 commits November 16, 2026 16:42
…d exact scope-check wording

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@TorecLuik

Copy link
Copy Markdown
Collaborator Author

Resolved in 34d9fa5: #97 (d0499b5) merged in, clean.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Import the Zarr contracts from biomero-schema and own the UI colours

1 participant